Picture Donald Knuth. Eighty-eight years old, the man who wrote The Art of Computer Programming by hand in TeX, which he invented himself to be able to write his own books. The father of algorithmic analysis. The most laudeated living computer scientist. A legend, and a well-known AI skeptic.

Now picture him reading a printed chat log between a fellow colleague and Claude Code. Not skimming. Reading it in detail, because there is something genuinely baffling about it.

The log belongs to Filip Stappers, a mathematician who ran thirty-one coding explorations with Claude Opus 4.6, systematically probing a class of combinatorial objects Knuth had spent decades thinking about. Exploration 15 surfaced something unexpected: a structural pattern nobody had written down.

Knuth read it, judged it valid, proved it correct by hand, and wrote a paper about it. He called it “Claude’s Cycles.” Knuth noted, with his characteristic precision, that he’ll “have to revise his opinions about generative AI” one of these days.

Most accounts that open with this scene take one of two off-ramps.

The first off-ramp is the replacement narrative. AI is now the scientist. The model had the insight; Stappers just ran the prompts; Knuth read the output and judged it true. Discovery has been automated. We are, depending on your temperature, either liberated or obsolete.

The second off-ramp is the stochastic-parrot dismissal. It’s just a language model predicting tokens. It doesn’t understand combinatorics; it doesn’t understand anything. Stappers did the science; Claude shuffled plausible-sounding symbols. Attribute the discovery to the researcher, not the autocomplete.

Both off-ramps feel satisfying. Both are wrong. And they’re wrong in the same way: they’re answering the question “did AI do the science?” That’s the wrong question. The interesting object is not the agent. It’s the loop.

The loop looks like this: a human poses a question; a model proposes candidates; a verifier filters the candidates; a human curates what survives. Round and round. What Stappers and Claude did is not fundamentally different in shape from what Tao and Lean are doing, or what the GNoME pipeline does in materials science, or what AlphaFold did for protein structure. The shape is the same. The loop does the discovery.

I want to be clear about what that means, because it’s easy to hear it as either a compliment to AI or a dismissal. It is neither. It’s an empirical claim about where the causal action lives. Not in the model, not in the human, but in the interaction structure between them. Get that structure right and you get science. Get it wrong and you get confident nonsense at scale. The details of what ‘wrong’ looks like are worth walking through carefully, and we’ll do that a couple of sections down.

But before we can dismiss either off-ramp, we need to walk four recent cases. Because in every one of them — Claude’s Cycles, Tao and Lean, AlphaFold, GNoME — the replacement narrative is not just philosophically confused. It is empirically wrong. The loop does the discovery.

Four Cases, One Shape

In each of the following cases, the loop does the discovery. I’m going to give you four of them across four domains, because I think the pattern only becomes undeniable when you see it that many times. Same shape, different materials.

Claude’s Cycles

You already have the scene. Stappers runs thirty-one explorations. Not one inspired conversation, thirty-one numbered, documented, methodical probes. Claude proposes. Stappers evaluates. Knuth reads the surviving logs, verifies the mathematical claims by hand, and writes. One human author. One credited co-explorer. The question was Stappers’; the verification was Knuth’s; the curation of which thirty-one explorations were worth pursuing was human throughout. The model was the proposer.

That word matters, and I want to be precise about it. Proposer. Not discoverer, not author, not scientist. The one that generates candidates fast enough that the verifier can find something in the haystack.

Tao and Lean

Terence Tao, Fields Medal, the current standard of living mathematical genius, has been publicly working through what it looks like to use LLMs for mathematical research. His account in the Notices of the American Mathematical Society in 2025 is careful and specific, and I appreciate the care because it’s easy to overread these things in both directions.

Here’s what you’re actually looking at: an LLM proposes proof steps, intermediate claims, candidate lemmas, reformulations of the problem. Lean’s type-checker is the verifier. And Lean’s type-checker cannot be fooled. It either accepts a proof term or it doesn’t. There’s no “plausible-sounding but wrong” in Lean. The system rejects garbage instantly and silently. What the human does — Tao, or anyone in his lab — is curate: which of the surviving proof steps are worth following, which directions are worth pursuing. The loop is tight. The proposer is creative and unreliable; the verifier is reliable and uncreative; the human operates between them.

I’d reframe that: the breakthrough isn’t that LLMs can now do mathematics. It’s that the loop can now run fast enough and cheaply enough to be useful.

AlphaFold

I want to include this one even though it’s older and better-known, because it’s the case where people most confidently say “AI solved the problem.” And that confidence is telling.

The problem was fifty years old. How does a protein fold? You have the amino acid sequence; you want the three-dimensional structure; that structure determines function; that function determines everything downstream in biology and drug design. AlphaFold 2 proposes structures. Experimental crystallography verifies them: X-ray diffraction, cryo-electron microscopy, techniques that require real equipment, real samples, real physics. You cannot hallucinate a crystal structure into being; the protein either folds the way the model says or it doesn’t, and the experimental method tells you which. AlphaFold 3 extended this to molecular complexes: DNA, RNA, small-molecule ligands binding to proteins.

I think this is worth sitting with: human researchers curated which structures were worth solving, which proteins mattered, which findings were publishable. AlphaFold proposed. Nature verified. Humans curated.

GNoME and A-Lab

This is the one I find most clarifying, because the verifier here is the most brutally physical. You can’t argue with a crystal.

GNoME — Google DeepMind’s Graph Networks for Materials Exploration, if you want the full name — generated 380,000 candidate stable crystal structures. That’s the proposal stage. Then A-Lab, UC Berkeley’s autonomous laboratory, took 58 of those candidates and actually tried to synthesize them. Robotically. With real chemicals, real furnaces, real diffraction equipment to check what came out. Forty-one novel materials were successfully synthesized in 17 days.

Think about the verifier in that loop. Not a type-checker, not a scoring function. A physical robot in a real lab mixing real compounds at real temperatures and asking whether the crystal forms. You cannot fake a crystal. If A-Lab says it synthesized a novel stable material, it synthesized a novel stable material. The model proposed; the physical world verified; human researchers curated which structures were interesting enough to attempt.

I want to be blunt here: the pattern is not subtle. In every case: model proposes, independent verifier filters, human curates. The model is never the verifier. The model is never the question-poser. The model occupies exactly one slot in the loop, and it’s the slot that benefits most from creativity, speed, and a high tolerance for being wrong.

The replacement narrative requires you to believe the model is doing all three jobs. It isn’t. Empirically, demonstrably, across four domains, it isn’t.

Before we anatomize what changed inside the loop, it is worth noting that this shape is fifty years old. Because that changes what we should find surprising.

The Loop Has Is Older Than You Think

The loop does the discovery today in the same way it did the discovery in 1976.

I want to be somewhat insistent about this, because I think the historical amnesia around computational assistance in science is part of what makes AI discourse so unmoored. This isn’t new. What’s new is narrower than you think, and understanding exactly what changed is the only way to correctly evaluate what it means.

Appel and Haken, 1976

You probably know this one: the four-color theorem says any map can be colored with four colors such that no adjacent regions share a color. Mathematicians had been trying to prove it since 1852. In 1976, Kenneth Appel and Wolfgang Haken proved it. Using a computer. Their proof involved reducing the problem to 1,482 configurations and verifying each one computationally. No human read every step. No human could. The proof was real; the verification was mechanical.

The mathematical community was genuinely unsettled. The theorem wasn’t wrong. But the proof didn’t fit the usual epistemological frame. You couldn’t follow it the way you follow a conventional proof. You had to trust the computer. That discomfort was the first serious confrontation with what I’m calling the loop: humans posed the problem, humans designed the reduction, a computer verified the 1,482 cases, humans accepted the surviving result.

Same shape. Fifty years earlier.

Hales and Flyspeck, 1998–2014

Thomas Hales proved Kepler’s sphere-packing conjecture in 1998. The conjecture: the way you’d intuitively stack cannonballs, face-centered cubic packing, is in fact the densest possible arrangement. The proof relied on computer enumeration so extensive that the referees couldn’t verify it. They were “99% confident” and said so, which is an unusual thing for mathematical referees to say.

Sixteen years later — sixteen years — the Flyspeck project completed a formal verification of the proof in HOL Light, a proof assistant. I want to dwell on that: sixteen years. The loop had a very slow verifier in the middle. It still worked. The result was real.

Sixteen years from claim to verified closure. I find that humbling. We talk about AI accelerating science as though “fast” is a new property of loops. The loop has always done the discovery. Sometimes slowly.

AI Feynman, 2020

You won’t usually see this one in the AI-in-science timelines, but it belongs here. Silviu-Marian Udrescu and Max Tegmark built a system called AI Feynman that uses symbolic regression (searching the space of mathematical expressions) to recover physics equations from data. Feed it measured relationships between physical quantities; it proposes the equation. Tested on 100 equations from the Feynman Lectures on Physics. Human scientists posed the problems; the system proposed expressions; formal mathematical checks filtered them.

This is 2020, two years before the moment people usually date as the AI-in-science inflection. I find it clarifying: the loop is the same.

What 2022 Actually Changed

Here’s what changed. One thing in the loop changed, and it’s the proposer slot.

Before roughly 2022, the proposer in loops like these was domain-specific, narrow, and hand-engineered. AlphaFold’s architecture was designed from the ground up for protein structure prediction. The Flyspeck enumeration was written for Kepler’s problem. AI Feynman’s symbolic regression engine was built for recovering physics equations. The verifiers were already strong: formal proof checkers, physical experiments, crystallography. The curators were already human. But building a proposer required significant domain-specific engineering effort for each new application.

What changed, and I think this is the key move, is that the proposer slot is now increasingly occupiable by general-purpose large language models that can be directed with natural-language specifications.

The same model that helps Stappers explore combinatorial objects can, with different prompts, propose protein structures, generate proof steps, suggest material candidates. It’s not that the model does these things well in some absolute sense. It does them well enough that a strong verifier can find the real results in the output. And FunSearch, DeepMind’s system for mathematical discovery, goes a step further: it uses LLMs to generate the search strategy itself. AlphaEvolve extends this to evolving the algorithms. The proposer writes the proposer.

I want to flag what didn’t change: verifier reliability, human curation, question-posing. Knuth still read the logs. Tao still decides which lemmas are worth pursuing. The IMO committee still wrote the problems. The physical world still decides whether a crystal forms.

So: what exactly is the loop, and what changed inside it? The proposer and verifier are different jobs — you can confuse them, and that confusion is the mistake that both off-ramps make.

Anatomy of the Loop

The loop does the discovery. But what is the loop, exactly? It has four roles: poser, proposer, verifier, curator. They are not interchangeable.

I want to be specific about each, because vagueness here is where confused takes come from. When someone says “AI did the science,” they’re usually collapsing all four roles into one and attributing them to the model. When someone says “it’s just a tool,” they’re usually denying that the proposer role is meaningfully distinct from, say, a search engine. Both collapses are wrong.

The Proposer Role

I want to define this carefully, because the proposer role is where most confusion lives. The proposer generates candidates. That’s it. It doesn’t need to know which candidates are correct; it doesn’t need to understand the domain at the level required to verify; it doesn’t need to bear accountability for wrong answers. It just needs to produce output that the verifier can evaluate. Fast. In large quantities. With enough breadth that interesting things appear in the distribution.

This is a real and important job. Generating candidates well, in a domain, at the right level of specificity, with the kind of variety that makes the verifier’s job tractable, is genuinely hard. What LLMs are good at is producing plausible candidates in natural-language-specified domains. That’s useful precisely because “plausible” is a different bar from “correct.” The proposer buys lottery tickets; the verifier checks which ones won.

Galactica, or: What Happens Without a Verifier

In November 2022, Meta released Galactica, a large language model trained on scientific literature. I remember when this dropped: the pitch was that it could reason about science, write papers, explain concepts, generate hypotheses. It was fluent. Confident. It had read more papers than any human alive.

It was retracted in three days.

Galactica produced plausible-sounding but fabricated citations. It generated chemistry that was wrong. It stated incorrect facts with the same calm certainty it used for correct ones. The model had no verifier. It had no external check against which its proposals were filtered. It was a proposer talking directly to readers who were treating it as a verifier.

That’s the failure mode. “LLMs are bad at science” is the wrong lesson. Galactica was a capable proposer deployed without a loop. Dump the proposer’s output directly into the world, skip the verifier, and you get confident nonsense. The problem wasn’t the model. The problem was the missing loop.

The Verifier Role

I want to be careful here, because this is where the asymmetry lives. The verifier is not creative. The verifier must be right. These are the two properties that matter.

Lean’s type-checker does not hallucinate. It processes a proof term and either accepts it or rejects it, and its answer is correct by construction: it’s checking against the formal rules of type theory, and it doesn’t have opinions or moods or bad days. A crystal either forms or it doesn’t. X-ray diffraction either confirms the predicted structure or it doesn’t. These verifiers are not AI systems. They’re physics, or mathematics, or formal logic.

Here’s a slogan worth keeping: the verifier is the one that matters. A loop with a weak proposer and a strong verifier still produces valid science — it’s just slow, because it needs more proposals before one survives — this is just what “normal science has been so far”, slow proposer, strong verifier.

A loop with a strong proposer and a weak verifier produces Galactica. The asymmetry is important. You can have crap proposals and still win, as long as the verifier is robust. You cannot have a weak verifier and still win, no matter how impressive the proposer.

Tao’s Insight on Composability

Tao articulated this in his 2025 Notices piece, and I think it’s genuinely the right way to think about it: the value of the LLM-plus-formal-verifier combination comes from combining complementary weaknesses. LLMs are creative but unreliable. Lean is reliable but cannot be creative; it needs to be told what to check. Individually, each is limited. Together, the loop covers both limitations: the LLM proposes, Lean certifies, and you get results that are both novel and guaranteed correct.

This is not a new insight in the philosophy of science or even in computational science. It’s the same structure as Appel-Haken: creative human mathematicians posed and structured the problem, mechanical verification checked the cases. What’s new is that the LLM makes the creative-but-unreliable proposer slot much cheaper and more general to fill. You don’t need a domain expert to hand-engineer each proposer; you need a prompt and a general-purpose model.

What’s Genuinely New Since 2022

Let me be specific, because this matters for evaluating claims. Three things changed:

First, open-ended program synthesis. The proposer can now write code, not just fill templates. FunSearch and AlphaEvolve don’t just suggest candidate solutions; they generate the search strategy itself, which then searches for solutions. The proposer proposes how to propose. That’s a qualitative shift.

Second, cross-domain transfer at useful fidelity. The same general-purpose model can propose protein structure candidates, proof steps, crystal candidates, combinatorial patterns, all with natural-language specification. You don’t need to rebuild the proposer for each domain. The moat that used to come from hand-engineering a domain-specific proposer is largely gone.

Third, tight LLM-to-formal-verifier loop latency. The Flyspeck project took sixteen years. The Lean loop that Tao describes closes in seconds. When the loop runs fast, you can iterate fast, which means you can attempt more ambitious problems and accumulate signal faster about which directions are productive.

What hasn’t changed: humans pose the problems; the verifier is still not an LLM; which questions are worth asking remains entirely human. Knuth read the logs. Tao decides which lemma is promising. The IMO committee wrote the problems. The GNoME team decided that stable crystal structures were the target. The question of what to look for has not been automated.

With the anatomy clear, we can finally steelman both off-ramps seriously. And explain precisely why they are both wrong.

Why Both Extremes Are Wrong

The loop does the discovery, but what does that mean for who gets the credit?

I want to take both positions seriously, because I think each is tracking something real. Dismissing either one without engagement is how you end up with a position that sounds crisp but falls apart in the edge cases.

The Maximalist Steelman

Knuth is eighty-eight years old. He has spent decades thinking about the class of combinatorial objects Stappers was exploring. He hadn’t found the pattern that surfaced at exploration 15. Stappers, presumably, hadn’t either, or there wouldn’t have been thirty-one explorations looking for it.

Here’s the maximalist claim: if surprise is constitutive of discovery, if discovery is the moment something genuinely unexpected becomes known, then the model contributed something essential. It’s not that the model was incidentally involved in a process humans could have completed. It’s that the specific result didn’t exist in any human mind before the model generated it. A system that reliably surfaces what experts miss is doing something scientists do. The fact that the model can’t feel pride or bear responsibility doesn’t settle whether it participated in discovery.

The maximalist is pointing at something real. The model wasn’t executing a search strategy that a human had pre-specified. It was navigating a space in ways that produced genuinely unexpected output. That matters.

The Dismissive Steelman

A hammer drives nails. We don’t say the hammer built the house. Excel executes arithmetic the analyst specified; we don’t credit Excel with the financial model. AlphaFold was trained on structural biology data that human researchers collected over decades; the training objective was designed by human engineers; the evaluation criteria were set by human scientists. The model is a sophisticated tool. Sophisticated tools don’t discover; they execute.

The dismissive case is not purely rhetorical. There’s a serious accountability point underneath it: when a model-generated finding turns out to be wrong (and some will), who is responsible? If you’ve attributed agency to the model, you’ve obscured the accountability chain. The human who deployed the loop without a strong verifier, who curated poorly, who published without checking: that human is responsible. Diffusing credit into the model diffuses accountability too, and that’s a practical problem, not just a philosophical one.

The dismissive steelman is also tracking something real. The model didn’t choose the question. It didn’t decide the result mattered. It didn’t design the verifier. It didn’t stake its reputation on the finding.

The Verdict

Both steelmans are right about something; both are wrong about the structure.

The maximalist is right that the model contributes something essential, something that wouldn’t have been there without it. Wrong that this constitutes independent discovery: you cannot have independent discovery without the ability to pose a question, which requires caring about the answer, which requires the kind of intentionality the model doesn’t have.

The dismissive is right that the model doesn’t choose questions, doesn’t bear accountability, and shouldn’t be treated as an author in the full sense. Wrong that this makes it a mere tool in the hammer-and-nail sense. AlphaTensor, DeepMind’s system that found faster matrix multiplication algorithms, didn’t just execute a search the engineers specified. It found an algorithm that reconfigured what experts believed was achievable. That’s not executing; it’s navigating a combinatorial space in a way that produces genuine surprise. The hammer never surprises you — except when you’re not looking and it hits your finger, which by the way, AI can also do, and with a louder bang.

The right frame is, I think, the AI lab member. Indispensable, capable, sometimes surprising. Not a hammer; not a principal investigator, not a replacement for another human. Just a genuinely, qualitatively new kind of entity. An entity that occupies the proposer slot in the loop and does it better than anything that occupied that slot before, but doesn’t touch the verifier, doesn’t pose the questions, and doesn’t bear the accountability that comes with authorship.

If AI is a lab member and not a scientist, what does that do to the publishable paper, and to the metrics we use to measure science?

But What Happens to the Paper

The loop does the discovery. But the paper still needs to be written, and the paper is what the institution counts.

This is where I get more pessimistic, or at least more cautious. The loop is good news for science-the-activity. It’s more complicated news for science-the-institution.

Discovery and Paper Count Were Already Decoupled

Park, Leahey, and Funk published in Nature in 2023 a careful empirical study of scientific disruption, measuring across decades of papers and patents how often new work displaces the prior literature versus consolidates it. Their finding: disruption has been declining since 1945. Not linearly, not dramatically, but consistently. Meanwhile, paper count has been exploding.

The interpretation I find most compelling: the processes that generate papers and the processes that generate genuine advances were already decoupled before AI. A lot of papers are small increments, confirmations, replications, applications of known methods to new datasets. That’s not waste; science needs that infrastructure. But it means paper count was already a noisy proxy for discovery rate.

What an AI proposer does to this: it makes generating candidate findings cheaper. Cheaper to generate means more candidates, means more papers. Discovery and paper count decouple further.

Goodhart as Accelerant

Goodhart’s Law: when a measure becomes a target, it ceases to be a good measure. Academic institutions have been targeting paper count, citation count, journal impact for decades. Paper mills, factories producing fake or low-quality research for pay-to-publish journals, were already a pre-AI problem. Ioannidis documented the replication crisis in 2005 and again in 2018; most published findings in some fields don’t replicate, and that was before AI made generating plausible-sounding results cheaper.

The AI proposer is an accelerant deployed into a system already optimizing for the wrong thing. It makes the Goodhart problem worse. A lot worse. If you can generate a thousand candidate papers in the time it used to take to generate ten, and the verifier in your loop is peer review (slow, inconsistent, and famously gameable), then you have a problem that is structurally different from the pre-AI problem in scale.

I want to be clear: this is not an argument against AI-in-science. It’s an argument that the institution of science has a verifier problem, and AI proposers make that verifier problem more urgent. The fix is not to slow the proposer. The fix is to build better verifiers.

The Optimistic Edge

Here’s where I land, and I’m genuinely somewhat optimistic about this: if generating candidates is cheap, the scarce skill shifts upstream.

The researchers who will matter most in this environment are not the ones who can generate the most proposals. They’re the ones who can pose the right questions, who can identify which problems are worth solving before they know the answer, and the ones who can build strong verifiers. Question-selection and verifier-design become the competitive moat. That’s a real and important skill. It’s harder to fake than generating a paper. And it’s the skill that the loop most depends on.

If I’m hiring for a lab in 2026, I’m not looking for “AI scientist” as a job description. I’m looking for people who can look at a domain and say: here is what a correct answer would have to look like, and here is how I would know if I found it. That’s what the verifier is. Building it is hard. It requires deep domain knowledge, epistemological clarity, and the kind of judgment that comes from years of thinking carefully about what you’re actually trying to know.

A Diagnostic Heuristic

Let me give you something practical. When you read an AI-in-science result, a press release, a paper, a breathless tweet thread, ask this: what was the verifier, and who built it?

If the verifier is Lean, or a crystal, or experimental replication with pre-registered protocols, or a physical measurement with known error bars: trust the result. The proposer’s reliability doesn’t matter much; the verifier caught the garbage. The finding is real regardless of how the candidates were generated.

If the verifier is not named, if the paper says “we used GPT-4 to generate hypotheses and evaluated them with GPT-4 to assess plausibility,” you’re looking at Galactica with extra steps. The proposer and the verifier are the same system. The loop is not closed. Be skeptical.

That diagnostic question is the institutional choice this moment is forcing. What was the verifier, and who built it? Let’s land it.

The Verifier Is the One That Matters

“I’ll have to revise my opinions about generative AI one of these days.”

Knuth said that. Not “AI did the science,” not “it’s just autocomplete.” Something more specific: I may have underestimated what this thing can do in the right loop. The loop did the discovery. The human owned the question and the verifier. The model was the proposer. And the result was real.

The Investment Gap

Here’s the uncomfortable institutional reality: almost all current investment is in the proposer. Larger models, more parameters, cheaper inference, better fine-tuning, faster generation. The race to make the proposer better is well-funded, well-publicized, and moving fast.

The verifier is comparatively neglected.

Formal verification tools like Lean exist, but they’re hard to use, require significant expertise, and don’t cover most scientific domains. Physical verification (A-Lab-style robotic synthesis) is expensive and slow relative to the speed at which the proposer can generate candidates. Experimental replication is underfunded as a scientific activity; it’s less prestigious than novel claims. The referee system in academic publishing was designed for a world where generating candidates was the hard part. It was not designed for a world where a model can generate ten thousand plausible candidates in an afternoon.

Hiring “AI scientists” misframes the institutional need. The need is for researchers who can pose hard questions and build reliable verifiers. The “AI lab member” frame points to what needs managing: not the model, but the loop. And the loop’s bottleneck right now is the verifier.

I should mention that this piece is drawn from the science chapter of Mostly Harmless AI, a book I’m writing about what AI actually does versus what the headlines claim. If you’re finding this useful, the book is where the longer argument lives.

What Comes Next

Next week: AI in creative output. Art, literature, music. Where the verifier question changes shape entirely. Because in mathematics and materials science, you can at least define what “correct” means. In creative domains, the verifier problem is not just harder. It’s constitutively different. What does it even mean for a creative proposal to be correct? I have some thoughts, and I think they’re going to make the science case look straightforward by comparison.

Stay curious.

P.S. There’s a Subscribe button somewhere below this. I’m told it does something useful. My understanding of subscription mechanics is below the level of a confident stochastic parrot. I believe it works, but I haven’t checked every step. Click it anyway. The verifier here is whether you keep showing up next Monday.

Subscribe now

Every post on the blog this month is on the theme of agent reliability, anchored on the second edition of Mostly Harmless AI — 50% off during early access — where the Limitations chapter walks all seven failure modes that compound into the curve below. You can also read the whole book online for free. More at the end.

The headline number you’ve seen on every AI-progress chart — “model X completes two-hour tasks half the time” — is the 50% reliability horizon. That number is moving fast. Doubling every seven months, per METR’s time-horizon work. It’s the curve on every AI-progress chart, the one conference talks lean on, the one that lands in investor decks.

The number that decides whether you can actually deploy an agent is a different one.

The 80% reliability horizon — the task length at which an agent finishes well enough that you would not feel the need to check — sits 70–80% below the 50% figure, and it moves up far more slowly. That gap is the difference between demo and deploy. The 50% is what passes the eval. The 80% is what survives the afternoon you weren’t watching. Not two hours. Thirty minutes you’d hand off.

I want to be precise about what I’m not arguing. I’m not arguing agents are broken or that AI progress isn’t real. It is real, and it’s fast. I’m making the narrower claim that the 50% and 80% horizons move at different speeds, and that the 80% is the one that matters when someone else’s data is on the line. This post is the math behind the gap, why it’s structural, and what you can do about it.

If you’re building on agents, you are building on the 80% horizon. The 50% number is for the marketing deck.

Subscribe now

Two horizons, two stories

METR’s methodology is clean: take a population of tasks with measured human completion times, then find the longest task a given model clears at success rate X. Do that for X = 50%, and you get the 50% horizon. Do it for X = 80%, and you get a different curve. A different story.

The 50% horizon has been doubling roughly every seven months. Late 2025, it sat around a couple of hours for software tasks. That’s the curve that makes headlines. That’s the curve you’ve seen on every slide. Striking.

The 80% horizon sits roughly 70–80% below. The same agent that clears a two-hour task half the time clears a half-hour task four-times-in-five. Not two hours. Thirty minutes. And that gap doesn’t close at the same rate. It moves slowly, stubbornly, for reasons that are mathematical before they are engineering.

So you have two curves growing at different speeds. The 50% horizon is the curve of capability: what can this system do, under ideal conditions, at least sometimes. The 80% horizon is the curve of trust: what can this system do reliably enough that you’d hand it a production key and walk away.

They are not the same curve. And they do not close the same way.

The longer your task horizon, the wider the gap between can-do-it-sometimes and can-be-trusted-with-it. The mechanical reason is one piece of math.

Probability arithmetic

Here’s the setup. A language model you’re calling spends a fixed compute budget per output token. Each step in a multi-step process has some per-step success probability p that is strictly less than one. The model is stochastic, the world is noisy, context degrades.

String n steps together, and the probability that all of them succeed is roughly pⁿ. That’s it. That’s the math.

Here’s what that looks like with actual numbers. Suppose your agent is excellent: p = 0.99 per step. That’s a 99% success rate on any single action. Compound it over 100 steps: 0.99¹⁰⁰ ≈ 0.37. You’ve gone from near-certain to worse-than-a-coin-flip without anyone making an error. Now drop to p = 0.95 (still quite good, still 95% per step). Over 100 steps: 0.95¹⁰⁰ ≈ 0.006. Six in a thousand runs succeed.

This is not a gap you close with next year’s training run. It is the shape of any probabilistic process operating in sequence over time. The curve doesn’t change when you improve p; it just shifts outward.

Reasoning models — the o-series, R1, extended-thinking variants — are valuable here. They buy you a higher per-step p, and they let you spend more steps at that higher rate. Both matter. But they push the curve outward. They do not change its shape.

Two pieces of evidence you should hold next to that math. GSM-Symbolic (Apple, 2024): perturb a math problem the model has seen (swap a name, change a number) and accuracy collapses even when the reasoning path is identical. The model has memorized the route, not the reasoning. Faith and Fate (Microsoft, 2023): transformer accuracy degrades with computational-graph depth even when each individual sub-step is solvable in isolation. Depth itself is the failure axis. More steps means more surface for p < 1 to accumulate.

Reasoning models buy you a higher per-step p and more steps to spend. They do not change the shape of the curve.

Where the chain gets long

Agents are exactly the setup that makes pⁿ painful.

Think through a typical agent run: read prompt, plan, call tool, read result, call tool again, critique output, adjust plan, call final tool, write response. Seven steps if you’re being generous. A real production agent reaches hundreds. Each step is one more p rolled. Each tool call is one more chance the orchestrator hands the tool the wrong arguments — garbage in, deduction out.

Self-critique doesn’t repair this — and you can verify the result yourself if you’ve tried it. Huang and colleagues (2024) showed that intrinsic self-correction without an external oracle signal actually degrades performance. The model talks itself out of correct answers as often as it talks itself in. The paradox is clean: if the model could recognize the error, it would not have made it. Asking it to introspect on failures is asking the broken compass to check itself.

So let’s put numbers on a real scenario. An agent that succeeds on each of five steps 95% of the time lands at 0.95⁵ ≈ 0.77. Decent. Not great, but workable. Now extend that same agent to a fifty-step trajectory: 0.95⁵⁰ ≈ 0.08. Eight runs out of a hundred finish correctly.

The demo ran five steps. The deploy runs fifty. The demo and the deploy are two different machines.

That’s the 80% horizon you’ll actually feel in production. It’s not a philosophical concern about AI reliability in the abstract. It’s the arithmetic of what happens when you take a stochastic generator and ask it to maintain a chain of reasoning over a long enough trajectory that pⁿ has time to do its work.

What you can actually do

Three mitigations. Each one genuine, and each one with a ceiling you should know before you commit.

Verifier-shaped tasks. Where the output can be checked deterministically (arithmetic, code that compiles and runs, SQL that parses, a formal proof), you can recover trust that the probabilistic generator alone cannot provide. A SAT solver beats an LLM on deductive closure every time. The architecture that wins here is LLM-proposes-candidate, deterministic-system-signs-off. The generator explores the space; the verifier approves the exit. This is, incidentally, the same pattern Monday’s post on the seventy-year argument named: a deterministic shell around a stochastic core, applied at the task level rather than the system level. The twist is that not every task has a fast verifier. Code that runs is checkable; code that runs correctly for all future inputs is not.

Retrieval-augmented generation. If the fact your agent needs is no longer arbitrary recall but lives in a curated document the model is required to cite, then Kalai and Vempala’s 2024 lower bound on calibrated hallucination does not apply to that fact. Most agent failures upstream of a tool call are recall failures the agent doesn’t know it’s making; retrieval changes the error mode from confident confabulation to visible gap. RAG turns a free-running generator into a paraphrase-and-summarize system over a known corpus. The reach of the system is now bounded by the reach of the index. But anything outside that index is back to pure p < 1 territory.

Narrow the horizon. The cheapest move is the one nobody wants to make: don’t deploy your agent on a fifty-step trajectory. Cut it to five. Hand off to a human at the boundary. At five steps with p = 0.95 you’re at 0.77; at fifty steps you’re at 0.08. That’s not a small difference. That’s the difference between a tool that works and a demo that occasionally works. Now, this trades autonomy for reliability. That trade is worth making in most production contexts right now. Whether it’s worth making in your context is a product question, not a research question.

Watching the right number

The 50% number will keep doubling and you should track it. That is real progress and worth watching closely.

But it is not the number your users feel. The number your users feel is whether the agent finished their task well enough that they didn’t have to re-run it, check its work, or clean up after it. The difference between “I tried that AI agent thing and it was magic” and “I tried that AI agent thing and it broke my Friday” is roughly the distance between the 50% horizon and the 80% horizon at your task length.

The shape of the next several years of agent engineering is already visible in the mitigations you’ll be reaching for: deterministic verifiers around stochastic generators, retrieval around recall, short trajectories with human handoffs where the math demands it. Not because agents are weak. They’re remarkable. But the pⁿ curve doesn’t care about benchmark scores. It cares about chain length.

One number, slowly creeping upward, every quarter. Watch that one.

Until next time, stay curious.

If the 80%-horizon framing landed, the second edition of Mostly Harmless AI walks the seven failure modes that produce the curve — the calibrated-hallucination lower bound, the U-shaped attention curve, the reversal curse, the depth ceiling on deduction, the rest. 50% off during early access. You can also read the whole thing online for free in a custom reader I built — dark mode, font controls, offline support, the works.

Get Mostly Harmless AI - 50% off

And if you want everything I’ve written, plus everything I’m going to write, that’s the Compendium. One purchase, in perpetuity.

Part of the run-up to the second edition of Mostly Harmless AI — 50% off during early access — where this is the spine of a new chapter on generative and multimodal AI. You can also read the whole book online for free. More at the end.

One morning, not so long ago, perhaps you asked Claude (or Gemini, or ChatGPT) to do something for you, and decided it was easier to just give it a picture of it than explain the whole thing. Perhaps it was “how do I cook this thing?” or “what building is that?” or “do this homework for me, please, please, my live depends on it”. Then you uploaded the picture, and back came a textual response.

Not happy with what the bot understood, you decided a thorough explanation was owed. But, alas, since all we got is a couple fatty fingers for typing, you decided it was best if you explained it with your own voice. And again, uhms and ehms notwithstanding, you again got a full response back, this time with an audio voice over.

Ten years ago, this simple dance of back-and-forth multimodal information would have required four separate research fields, each with its own conferences, its own vocabulary, and its own priesthood. They have quietly become one single thing. It’s all tokens all the way down. Language has subsumed all modalities. This is how.

Subscribe now

The recipe never cared what it was eating

Strip “generative AI” down to the one idea doing all the work and you get a single sentence: look at a big pile of examples, learn the distribution that produced them, then draw new samples from it. That is the whole trick. It is what a language model does, and it is the only thing a language model does. It is also what an image model does, and an audio model, and a video model.

The recipe is indifferent to what the examples are. Text is a one-dimensional run of symbols. An image is a two-dimensional field of colour. Audio is a pressure wave sampled tens of thousands of times a second. Video is all of that, plus time, which is why it is the hardest. Four different shapes of data, one identical question asked of each: given what I have seen so far, what plausibly comes next? The machinery that answers that question does not need to know whether “next” means a word, a patch of pixels, or a slice of waveform. It only needs the data turned into a sequence of countable things.

Tokens.

So the thing we have been calling a language model was never really about language. Or, put better, it was never about written language. It turns out, language is something far more powerful.

Ask any linguist, and they’ll say any set of sequences of distinct symbols (tokens) can be modelled as a language. It doesn’t matter what your symbols are—letters, words, patches of images, numbers in a math formula, whatever—language is just the structure around them, what makes some sequences valid and others nonsense.

This is the key idea. All else is (incredibly good) engineering.

A decade of building the same machine, separately

It did not look that way while it was happening. For about a decade every modality got its own bespoke contraption, and each one looked like its own discipline.

Image people had generative adversarial networks: a forger and a detective locked in a training duel, the forger getting better at faking until the detective could no longer tell. The beautiful idea buried in there — and the one that survived the technique itself — was the latent space: a compressed interior map of “all possible faces,” where walking in a straight line morphs one plausible face smoothly into another. GANs were temperamental, prone to collapsing into a single good fake and refusing to leave, and by the early 2020s they had lost the lead. The latent-space intuition outlived them and runs underneath everything that came after.

Then diffusion took over image generation with a trick that sounds like it shouldn’t work. Take a real photo, add a little static, add a little more, keep going until it is pure snow. Now train a network to undo one step of that. To make a new image, start from snow and run the undo, over and over, until something coherent surfaces. It is sculpture by removing noise instead of removing marble, and it is what powers essentially every image generator you have used.

Audio had its own separate lineage: speech-to-text built one way, text-to-speech another, music a third. Text had the large language models, off in their own enormous-budget corner of the field. Four communities, four sets of architectures, four sets of war stories. If you had asked, in 2021, whether the image people and the language people were building the same machine, both sides would have laughed.

CLIP quietly knocks out the wall

The crack in the wall came from a 2021 model whose job sounds almost too modest to matter: teach one system that the word dog and a photograph of a dog are talking about the same thing.

The way you do that is to train a text encoder and an image encoder together, on hundreds of millions of caption-and-picture pairs, with one instruction: put a picture and its true caption close together in a shared space, and shove mismatched pairs apart. What you get at the end is a single space where “a photo of a golden retriever” and an actual photo of a golden retriever land as neighbours. Text and pixels, in the same room, with the same coordinates.

That sounds like a party trick for image search. It was the hinge the whole field turned on. Once text and images live in one space, text can steer image generation — point the diffusion process at the region of the space that means “golden retriever in a spacesuit,” and let it denoise toward there. Every text-to-image system you have used is, under the paint, that move. And the deeper implication was harder to ignore than the application: if you can put two modalities in one space, the wall between them was never structural. It was just a wall nobody had walked through yet.

Tokens all the way down

Here is where it lands. By the mid-2020s the bespoke machines stopped being separate machines.

The move is almost embarrassingly direct. Tokenise everything. Text already broke into tokens. Cut an image into a grid of patches and treat each patch as a token. Run audio through a neural codec that emits discrete chunks, and those are tokens too. Now you do not have a text stream and an image stream and an audio stream. You have one stream of tokens that happen to have come from different alphabets, and you train a single model on the only objective that was ever in play: predict the next token, whatever kind it is.

A model trained that way reads and writes everything, because to it there is no “everything” — there is just the sequence and the next position in it. You have used these. The one that holds a spoken conversation with sub-second latency, looks at the photo you paste in, and writes you a paragraph back is not a language model bolted to an image model bolted to a speech model. It is one model that was never told these were different problems.

Which is why the question that organised the field for a decade — is this a language model or an image model? — has quietly stopped having an answer. It is the same machine. The only thing that ever changed between text and pixels and sound was the alphabet, and the transformer emitting the next token has never cared which alphabet it is spelling in. It is tokens all the way down. “Language modelling” was a local name for something with no allegiance to language at all: modelling sequences of anything we can count.

The honest part

It would be easy to end on the astonishment, and the astonishment is real. One model, every modality, falling out of one stubbornly simple objective applied to a wider and wider definition of “token” — that is one of the genuinely beautiful results of the decade, and the kind of unification that does not come along often.

But unification is not the same as understanding, and I am not going to let the elegance smuggle that past you. A system that can place “dog” next to a dog in its latent space has learned the statistics of how dogs are described and depicted.

Whether it has learned what a dog is is a different question, and the convergence story does not answer it. It just makes the question apply to every modality at once instead of only to text. The machine got more general. It did not get more grounded necessarily. Both of those are true at once, and the interesting work of the next few years lives in the gap between them.

Until next time, stay curious.

This is the core argument of a new chapter in the second edition of Mostly Harmless AI — the full chapter walks GANs, diffusion, CLIP, audio, and native multimodality with the scenes and citations this post had to cut, and it is 50% off during early access. The whole book is also free to read online. If you want the rest of the argument — how these systems are trained, where they break, and what to actually do about it — that is what the book is for.

Get it (50% off) for life

You type a message to an AI assistant and it answers. The answer isn’t looked up. It isn’t scripted. The model generated it, character by character, guided by a single mathematical question: what comes next?

That question is the foundation of every language model ever built. A language model is a probability distribution over text — a function that, given a sequence of words, assigns a probability to every possible continuation. “The cat sat on the mat” scores higher than “the mat sat on the cat” not because a language model understands what cats do, but because the first sequence appears in human text and the second doesn’t. The model has compressed the co-occurrence patterns of an enormous corpus into its weights, and that compression is what produces a score.

Generative: you give the model a prefix, it samples the highest-probability next token, appends it, samples again, and repeats until a stop token arrives. The multi-paragraph response you got from ChatGPT this morning was that loop running a few hundred times. No lookup table. No if-else tree. No pre-scripted answers. Just: given all of this text, what is most likely to come next?

Here’s the thing I find quietly strange about this: it works. A procedure this simple — assign probabilities, sample the maximum — has produced the most influential technology of the last decade. What makes it work isn’t the procedure. It’s everything that goes into building a probability distribution that’s actually good. Good enough to write coherent paragraphs. Good enough to reason about code. Good enough to pass the bar exam and explain quantum mechanics in language your parents can follow.

Getting there took decades of compounding ideas. The arc is what this piece covers — from the crudest possible approximation of “probability over text” to the current frontier, where models are learning to think.

Each section of what follows is best understood as a response to the failure of the previous one. N-gram models worked until they didn’t. Neural embeddings fixed the part that broke. Pretraining scaled the fix to the size where it became genuinely impressive. Instruction tuning made the result useful for the first time. Preference learning fixed what instruction tuning couldn’t. Reasoning models added something nobody was sure was trainable at all.

Seven steps. One direction.

Every post on the blog this month is on the theme of agent reliability, anchored on the second edition of Mostly Harmless AI, where the engineering details that don’t fit a blog post live. You can also read the whole book online for free in a custom reader I built. More at the end.

This post may be truncated in your email. Read it online for the best experience.

Subscribe now

The n-gram intuition

The simplest possible implementation of “probability over text” is a lookup table.

Take a large corpus — a hundred million words will do to start. For every three-word sequence (trigram) in that corpus, record which word follows it most often, and with what frequency. “The quick brown” → “fox,” nine times out of ten, because Project Gutenberg is full of that particular sentence. “The capital of” → “France” thirty percent of the time, “Germany” twelve, “England” eleven, and so on through the geography. For every trigram you’ve seen, you have a probability distribution over what comes next.

Now generate text. Give the model “The quick brown” and it continues: “fox.” Give it “jumped over the” and it continues: “lazy.” Three words in and it’s generated “The quick brown fox jumped over the lazy” — and if you’re lucky, it lands “dog” and you’ve reproduced a famous sentence entirely from corpus statistics. Locally, it’s plausible. You could read a sentence of this and not immediately know you’re looking at a machine.

The problems start fast. By the third sentence, the model has no idea it was talking about a fox. It only remembers the last three words. You ask it to continue “The fox had been running from the” and it has no idea that a fox is involved, or that running happened, or that there’s a pursuit in progress. It just has three words and a lookup table. The output is statistically English. It is not coherent.

This is the Markov assumption: the next word depends only on the last N words, not on the full history of the text. For N=3, it’s a trigram model. You can increase N — five-gram models were standard in commercial speech recognition for years — but the table explodes. Fifty thousand words in the English vocabulary gives 50,000³ possible trigram contexts, roughly 125 trillion entries. At N=10, the number of possible sequences exceeds the estimated number of atoms in the observable universe. The table can never be complete enough to cover the distribution.

There’s a real engineering solution to the “we haven’t seen this exact trigram” problem: smoothing and interpolation. Estimate the probability of an unseen N-gram from shorter sub-sequences. Hidden Markov models formalised this in a probabilistic framework that, by the 1990s, had enough polish to power industrial speech recognition and early machine translation. I don’t want to undersell it — it worked. It was genuinely useful. It just topped out.

The wall is fundamental. Real language has dependencies that can be arbitrarily long. “The man who sold the car that had been parked in front of the house where my grandmother lived was finally found.” The subject of “was finally found” is seventeen words and three nested clauses back. No N-gram model reaches it. You need something that can condition on the full context — or at least compress the full context intelligently — rather than amnesiacally forget everything more than N words ago.

You need a model that generalises from sequences it has seen to sequences it hasn’t. A lookup table can only interpolate from what it’s seen before. What you need is something that has understood the pattern deeply enough to extrapolate.

N-gram models work until they don’t — and they don’t beyond a few words.

Words as numbers

Neural networks can learn the compression n-gram models can’t achieve. But they have a hard prerequisite: they operate on numbers. Words are symbols. Before a neural network can do anything useful with text, you need to represent words as vectors. The naive approach throws away everything that matters.

The obvious first attempt is one-hot encoding. Vocabulary of 50,000 words; each word is a vector of length 50,000 with a single 1 and 49,999 zeros. “Cat” is at position 4,312; “dog” is at position 17,846; “carburetor” is somewhere else entirely. The problem: nothing in this representation suggests that “cat” and “dog” are more similar to each other than either is to “carburetor.” The distance between every pair of one-hot vectors is identical. You’ve handed the network a symbol system with no structure, and it has to reconstruct the structure from scratch — spending enormous capacity learning that cats and dogs are both animals, that both appear near “fur” and “vet,” that “cat food” and “dog food” are structurally related — before it can learn anything about how language actually works.

The key insight that resolved this came from linguistics, not machine learning, and I think it’s underrated as an idea. J.R. Firth, writing in 1957: “you shall know a word by the company it keeps.” The distributional hypothesis. Words that appear in similar contexts — near similar neighbouring words, in similar grammatical positions — tend to have similar meanings. “Cat” and “dog” both appear near “pet,” “feed,” “vet,” “owner,” “fur,” “collar.” The context is a fingerprint of the meaning. Encode that fingerprint in a vector and you have a representation where similar words land close together in space.

Word2Vec (Mikolov et al., 2013) turned this into a training procedure. Train a shallow neural network to predict a word from its surrounding context words, or vice versa. Force each word’s representation down into a dense vector of, say, 300 floating-point numbers. Train on a billion words of text. The network learns that words appearing in similar contexts should have similar representations, because that’s what makes the prediction task cheaper. Words with similar distributional patterns end up with similar vectors — not because anyone programmed that, but because it follows from the objective.

The result that made people pay attention: vector arithmetic encodes semantic relationships. Take the vector for “king,” subtract the vector for “man,” add the vector for “woman.” The nearest vector in the resulting space is “queen.” Paris minus France plus Italy is approximately Rome. Try it yourself: it works because the structural relationship between “king” and “queen” is parallel to the relationship between “man” and “woman” in how the four words co-occur with everything around them. No one wrote these analogies in. The geometry of the space mirrors the structure of meaning, because both are implicit in how words appear together in natural language.

I find this genuinely strange, in the best possible way. You trained a network to do a simple word-guessing task on flat text, and the side effect was an algebra of concepts. The geometry was always latent in the co-occurrence patterns. Word2Vec just made it legible.

Modern language models don’t use Word2Vec as a separate preprocessing step — the embedding representations are learned jointly with the rest of the network during training on text. But Word2Vec’s intuition is why learned embeddings work at all. Once language is geometry, gradient descent has a surface to grip. You can compute distances, optimise them, stack arbitrarily deep networks on top, and train the whole thing end-to-end.

Embeddings are how we lie to neural networks in a useful way. We pretend words are points in space, so the math works out.

Pretraining

Now scale it.

Take a deep neural network — not the shallow two-layer thing in Word2Vec, but a transformer with dozens or hundreds of layers, billions of parameters, and an attention mechanism in every one of them. Feed it next-token prediction across the entire accessible internet: Wikipedia, GitHub, every book ever digitised, every forum thread, every research paper, every recipe, every political argument, every user manual for every piece of machinery ever manufactured. Same objective the n-gram model had: given what came before, what comes next? Except now the model has billions of parameters to compress the patterns into, the training signal is trillions of tokens, and the architecture is built to handle arbitrarily long context.

The architecture is what made everything else possible. The transformer (Vaswani et al., 2017) uses self-attention as its core operation. For each token in the input, self-attention computes relevance weights over every other token in the sequence — learned weights, computed from the data, different for each token, different in each layer. A pronoun can attend strongly to the noun it refers to, twenty positions back. A closing argument can reach back to the premise from the opening paragraph. There is no fixed window; the model considers, in principle, the full context at every step.

This is what broke the n-gram scaling wall. Not a bigger lookup table. Not smarter interpolation. A learned, flexible attention mechanism that compresses long-range dependencies into the model’s weights rather than trying to enumerate every possible context sequence. The key property, and it’s the one I keep coming back to: soft. Self-attention doesn’t pick one relevant token; it blends all of them with learned weights. The whole sequence contributes to every prediction, with a learned notion of how much each part matters.

The other critical property is self-supervised learning. There are no human-provided labels anywhere in pretraining. The text itself is the training signal. Show the model “The capital of” and ask it to predict “France.” It’s wrong; the gradient flows; the weights update. Show it three trillion tokens; let the gradient flow three trillion times. The entire digitised corpus of human knowledge is your training set, with zero labelling cost, because the next token is always right there.

Kaplan et al., 2020 measured loss as a function of model size, dataset size, and compute over seven orders of magnitude. The result: loss falls as a clean power law across all three dimensions. Double the parameters, get a predictable drop in loss. Double the training data, same. Scale is not a bet on something uncertain; it is a known return on investment, measured and re-measured across a staggering range.

Hoffmann et al., 2022 — the Chinchilla paper — corrected a real error in how the field had been applying Kaplan’s result. Most large models of 2020-21 had been trained on far fewer tokens than their parameter count justified. The Kaplan result suggested scaling up models as fast as possible; Hoffmann’s finding was that you need to scale model size and training data together, roughly equally, for a given compute budget. A 70B-parameter model trained on 1.4 trillion tokens outperformed a 280B-parameter model trained on fewer tokens at the same total compute cost. Smaller model, more data, better result. Scale both dimensions together.

What do you get at the end of all this? A base model. And this is the part that surprises people who haven’t seen one.

Type “What is the capital of France?” into a raw pretrained model and it continues the text. Maybe it writes out a geography quiz — “What is the capital of France? What is the capital of Germany? What is the capital of Italy?” Maybe it generates a fake Wikipedia article. Maybe it starts a trivia show transcript. It has not answered your question. It has found the most probable continuation of your prompt, given everything it absorbed during training.

The base model has absorbed more text than any human could read in a thousand lifetimes. The co-occurrence patterns of the entire digitised corpus of human writing are in those weights. It knows facts, relationships, styles, concepts, code, chemistry, poetry, legal prose, and every other form in which humans have arranged words.

It was trained to continue, not to respond. Ask it a direct question and it treats the question as the opening line of some text pattern — one it will extend in whatever direction seems most probable. It has no concept of “you asked me something and I should answer it.”

Pretraining gives a model knowledge. It does not give it manners, opinions, or any idea what you want from it.

Instruction tuning

Step one of making a base model useful: show it what “useful” looks like.

Collect thousands of demonstration pairs. A human writer sits with a prompt — “Explain the difference between supervised and unsupervised learning in plain English,” “Write a polite email declining this meeting invitation,” “Debug this Python function” — and writes the ideal response. Then fine-tune the pretrained base model on these (prompt, response) pairs using the same next-token objective, now applied to curated demonstrations rather than the open web.

This is supervised fine-tuning, or SFT. It is plain supervised learning — the same paradigm that has been in the machine learning textbooks since the 1980s. What’s new is only what it’s being applied to.

The headline result from Ouyang et al., 2022 — the InstructGPT paper — is still worth stating plainly: a 1.3 billion-parameter model, fine-tuned on human-written instruction-following demonstrations, was preferred by human evaluators over a raw 175 billion-parameter GPT-3. One percent of the parameters. Preferred.

Sit with that. The quality of the training signal matters more than raw scale. A carefully curated set of demonstrations of what “helpful answering” looks like is worth more, for the specific goal of being helpful, than a hundred times more parameters trained on unstructured internet text. The base model knows more. The instruction-tuned model is more useful. These are different things.

SFT teaches the shape of a helpful answer: addressed to the question asked, reasonably structured, proportionate in length, appropriate in tone. These are learnable patterns. The base model already has all the relevant knowledge in its weights; SFT is teaching it to retrieve and present that knowledge in a particular format.

Here’s the failure mode, and it matters for understanding everything that comes next.

SFT shows the model what good answers look like. It gives no mechanism for the model to evaluate, at generation time, which of two candidate continuations is more accurate, more honest, or less likely to cause harm. The model learned to imitate the shape of correct answers; it did not learn to prefer correctness over fluency when the two conflict. A confidently phrased wrong answer and a confidently phrased right answer can look identical from a format standpoint. SFT cannot distinguish them.

Teaching consistent refusals is especially brittle. To get a model to reliably refuse a class of harmful requests via SFT, you need human-written refusals for every phrasing variant you can anticipate. You will miss variants. The model has no general theory of harm. It has only pattern-matching against the phrasings it saw. Change the phrasing, add a fictional framing, ask in a different language, and the refusal can fail.

The deep limitation is this: SFT can teach what a good answer looks like, but it cannot teach which of two candidate answers is better. For that, you need to know something about better that you didn’t encode in any single example. You need preferences.

SFT teaches the shape of a good answer. It has no way to choose between two good shapes.

From demonstrations to preferences

The move that follows from SFT’s failure: what humans can do faster than writing demonstrations is ranking them.

Show a rater two model responses to the same prompt — response A and response B — and ask which is better. They can answer in seconds. Writing a response from scratch takes minutes. This means you can collect preference labels at much higher volume than demonstrations, and the preference label contains a different kind of information: not “here is the target,” but “this is closer to the target than that.”

Scale up the preference collection. Collect hundreds of thousands of (prompt, response A, response B, human ranking) tuples. Train a small auxiliary model — the reward model — to predict the human rankings: given a prompt and a response, output a scalar score. Then use reinforcement learning (specifically PPO) to push the language model toward generating responses the reward model scores highly.

This is RLHF — reinforcement learning from human feedback. Ouyang et al., 2022 used it as the third stage of the InstructGPT pipeline: pretraining → SFT → RLHF. ChatGPT’s characteristic tone — helpful, reliably cautious about harmful requests, good at hedging uncertainty, consistent about refusals — comes almost entirely from this stage.

The deepest shift over SFT: the model now has feedback about direction, not just target. SFT says “produce something like this example.” RLHF says “of the things you just produced, this kind is better than that kind — adjust accordingly.” A direction is a richer signal than a target. It can propagate to novel situations no demonstration ever covered.

The practical problem is that RLHF is a genuine engineering challenge. PPO is unstable. The reward model can be gamed: the policy learns to produce outputs that score highly on the reward model without actually being better, because the reward model is an imperfect proxy for true quality. Over long training runs, the policy finds exploitable features in the reward model and optimises for those rather than for what humans actually wanted. Balancing the RL update against the pretrained base (so the model doesn’t drift into incoherence while chasing reward) requires careful tuning. RLHF works, but it’s expensive, brittle, and hard to reproduce without a dedicated ML infrastructure team behind it.

Rafailov et al., 2023 found something that, in retrospect, looks almost obvious: you can skip the reward model entirely.

The paper, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” makes a mathematical observation. The preference-fitting problem that RLHF solves via a reward model + PPO can be reformulated as a classification loss directly on the language model policy. Given a preferred response and a dispreferred response to the same prompt, you want the model to be more likely to produce the preferred one. You don’t need a separate reward model to express that preference. You don’t need PPO to optimise it. The preference is a loss; the loss can be minimised directly on the policy.

DPO is computationally lighter and far easier to get working. The abstract says it “eliminates the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning.” In practice, the gap between “has the resources of a frontier lab” and “fine-tuned a model that actually behaves well” narrowed substantially after DPO. Within a year of publication, it became the default alignment approach for most open-weight instruct models — the Llama family, Mistral, Qwen, and most of their derivatives. When you read that a model is “instruction-tuned” in 2024 or later, DPO or one of its descendants is almost always in the pipeline.

Both RLHF and DPO require human preference labels. In 2022-23, this was tractable. By 2024, at frontier scale, it was a real bottleneck. You need raters capable of judging quality on maths, code, medicine, science. You can’t hire enough such raters to keep pace with the rate at which models can generate candidate outputs.

The field’s response was predictable in retrospect: use the models themselves.

RLAIF (Lee et al., 2023) replaces human raters with a strong language model as the preference oracle. Head-to-head comparisons with RLHF showed that AI-generated preference labels are competitive with human ones on summarisation and dialogue. The reward model trained on AI labels performs comparably to the one trained on human labels. The human rater is no longer in the loop.

Constitutional AI (Bai et al., 2022, Anthropic) does something more principled. Write a list of principles — a constitution — stating what the model should and shouldn’t do. Ask the model to critique its own outputs against those principles and revise them. The critiques and revisions become training data. The RL stage uses the model’s own evaluations as the reward signal. Human preference labelling is replaced by explicit normative reasoning: the model has to argue about whether its outputs satisfy the stated principles, not just produce outputs that pattern-match to human-labelled examples.

The logic extends one step further. If models can generate reliable preference labels, can they generate training data directly? By 2024-25, the answer in a widening range of domains was yes. Maths problems with worked solutions. Code problems paired with passing test suites. Instruction-following demonstrations written by large models to train smaller ones — the distillation pipeline, where a 70B model generates training examples that improve a 7B model, and the better 7B model feeds the next iteration. By 2025, a substantial fraction of the data used to train frontier models isn’t scraped from the web. It’s generated by earlier versions of the models themselves.

Ilya Sutskever, speaking at NeurIPS 2024: “Pre-training as we know it will end. Data is the fossil fuel of AI. We have but one internet.” The scaling curve that had defined the field since Kaplan 2020 was visibly flattening. The field didn’t slow down. The growth frontier moved: from bigger pretraining to better post-training. The headline AI announcements of 2024 were not “we trained a bigger model on more of the internet.” They were “we trained a better model by using our previous models to generate, evaluate, and curate the training signal.”

The sharpest shift in AI in 2023-24 wasn’t a bigger model. It was figuring out how to use models to train better models.

A new axis

There is a 2024-25 discovery that changes the picture in a qualitatively different way. Not a refinement of post-training preference optimisation. Something new.

RL doesn’t just align models. It can teach them to think.

The observation that sets it up: language models already have scratch space. Their output is text; nothing prevents them from writing intermediate reasoning steps before writing a final answer. Chain-of-thought prompting — asking a model to “think step by step” — has been known since 2022 to improve performance on reasoning tasks. The model writes out intermediate steps, and those steps help it arrive at a better final answer.

But chain-of-thought as a prompting technique has a persistent problem. The intermediate steps are generated by the same forward pass as the final answer. You can ask the model to think out loud, but you can’t verify that the scratchpad is doing reasoning work rather than performing reasoning for the reader. A model that writes plausible-sounding intermediate steps that happen to be wrong, then arrives at a wrong final answer, has not improved by being asked to show its work. The steps are decorative.

The o-series models from OpenAI in late 2024 made a conceptually simple training move: use RL where the reward is the correctness of the final answer, and leave the intermediate chain of thought entirely unsupervised. The model can write whatever it wants in the scratchpad. The only signal is whether the final answer is right.

What emerged from training was not what anyone programmed in. The model learned, without any explicit supervision of the intermediate steps, to use the scratchpad as actual working memory. Backtracking when an approach failed. Trying alternate formulations when one hit a wall. Verifying intermediate results before continuing. Restarting from scratch, several steps back, when it found an error in something it had already written. None of these behaviours appeared in labelled training examples. They fell out of the objective: over enough RL iterations, the training process found that careful scratchpad use led to more correct final answers, and it reinforced that.

DeepSeek-R1 (arXiv:2501.12948, January 2025, open weights) replicated the result outside a closed lab. Pure RL on reasoning trajectories; no human-labelled chains required. The paper documents what they call the “aha moment” in training: a specific point where the model spontaneously began verifying its own intermediate steps and restarting when they failed. Not because the training data contained this behaviour as a pattern. Because the reward for correct final answers made careful intermediate reasoning instrumentally useful, and the RL training loop discovered it.

I think DeepSeek-R1 is the clearest published demonstration of something the field had suspected but hadn’t proven at scale: that reasoning, as a behaviour, is trainable from a simple outcome-based reward signal. You don’t need human annotations of good reasoning traces. You don’t need to supervise the scratchpad. You need to reward the right answer and run enough RL. The reasoning emerges.

Three things follow from this that are worth naming separately.

Test-time compute is a new scaling axis. Pretraining scales with more data and more parameters — you pay at training time and get a more capable model. Reasoning models scale with more inference compute — you pay at generation time, by thinking longer, and get a better answer on the current problem. A smaller reasoning model that thinks for ten seconds can match or outperform a larger standard model answering in one pass. These axes are complementary, not competing. You can now trade training-time capability against inference-time deliberation, and that tradeoff is explicit and controllable in a way it wasn’t before.

Diagnosability changes the failure mode. A standard model that gets a maths problem wrong gives you a wrong number. A reasoning model that gets it wrong gives you a chain of thought — readable, traceable, inspectable at every step. You can see exactly where the logic went off course: which intermediate claim was false, which inference was unwarranted, at what point the reasoning was solid and where it broke down. For systems where the reliability of the output matters — and in agent pipelines, it almost always does — this is the property that makes the difference. The failure is visible. Visible failures are debuggable. Black-box failures are not.

And the arc closes. The whole story of this piece — n-grams, embeddings, pretraining, instruction tuning, preference learning, and now reasoning — is one continuous story of making the training signal more specific. N-gram models encode raw co-occurrence statistics: this is what tends to follow that. Embeddings compress those statistics into geometry that neural networks can use. Pretraining scales that compression to the entire digitised corpus of human writing. Instruction tuning adds: here is what a helpful answer looks like. RLHF and DPO add: here is what better looks like, relative to what you just produced. RLAIF and synthetic data close the loop so models can teach each other. Reasoning models add the final turn: here is what thinking carefully looks like. Not by showing examples of good reasoning. By rewarding the right final answer, and letting the model figure out the rest.

Reasoning models aren’t smarter than other models. They’re models that have learned to spend their intelligence more deliberately.

Each step in this story makes the feedback signal richer. Each step exists because the previous step’s signal wasn’t specific enough.

The direction is clear: we keep finding more precise ways to tell models what we want, and they keep using it.

Until next time, stay curious.

The second edition of Mostly Harmless AI goes deeper on what these training paradigms make possible in practice — why a reasoning model behaves differently as an agent core, what alignment actually means when you’re building a system rather than evaluating a benchmark, and the chapters that didn’t fit any blog post. The whole book is also available to read online for free in a reader I built and am rather fond of: dark mode, font controls, progress tracking, offline support, the works.

If you want everything I’ve written and everything I’m going to write, the Compendium bundles it all — one purchase, in perpetuity.

Check it out

You’ve been using AI coding agents for months. You’ve crafted elaborate system prompts. You’ve added a dozen skills. You’ve learned the dance of context window management. And somewhere around the third hour of work, something breaks. The agent starts forgetting things. Making wrong assumptions. Doing something close—but not quite—what you asked.

This isn’t a failure of the model. This is a failure of the system.

To be sure, better models make things easier. And models are getting better by the day. But no matter how good a model is, bad systems lead to bad outputs. Even the smartest people produce junk when fed with incorrect assumptions or given incomplete instructions.

In contrast, a good system with clear boundaries and explicit rules, that leaves the exact amount of flexibility necessary, makes creativity and productivity thrive.

You see this day and night in teams (of real humans) in every industry. It’s not often the smartest person in the room that solves the hard problem. It’s when you combine the right kinds of intelligence with the right kind of system that things click.

In this article, I want to make the case for a structured way to think about Large Language Model (LLM)-based agentic systems (mostly for coding, but also for knowledge work in general) that fixes some of the greatest pains I (and I sure most of you) have been facing when trying to scale AI-assisted workflows to professional levels.

It’s a system that puts the right constraints in the right places and leaves just enough space for creative exploration (or however you want to call what LLMs do when they hallucinate in your favor). It’s also a system that makes it clear you are in charge.

Everything an AI agent does happens inside a context window. System prompt, user input, tool results, skill injections—they all live there. The agent’s only mechanism for action is the ReAct (Reasoning + Acting) loop: think, call tools, observe results, repeat. Each cycle grows the context. Each skill activation injects more.

This creates a fundamental tension: context is power, but context is finite. Too little and the agent can’t connect the dots. Too much and the important stuff drowns. The gap between those two failure modes is narrow—and most agent frameworks ignore it entirely.

I’ll walk through why current systems fail, introduce a four-element framework for thinking about agentic architectures, show you how these principles apply across three domains, then present a vision for better AI harness engineering.

Part I - The Symptoms

To understand the problems we first need to understand how a standard agentic loop works. The typical architecture is what’s called a ReAct loop. The LLM runs in a loop that determines the next action given context, which can be read some files, ask the user, invoke a tool, inject a skill, etc. When the agent decides no more actions are necessary, the loop ends and the user is given control back to continue the prompt.

That’s it. All the seemingly supersmart behaviours of Claude Code, Gemini CLI, and Codex are, under the hood, some form of the basic ReAct loop. There are of course nuances. For example, most systems decide that if the agent calls the same tool with the same args three times, it must be stuck in a loop and stop the turn. There are perhaps hard limits on how many tool calls the agent can do in each turn.

Context is the bottleneck. Not the model. Not the prompt. Context.

The agent doesn’t have memory. It doesn’t have state. It has context. Everything it knows about your project, your preferences, your conventions, all of it lives in the context window. When you add a skill, you’re injecting more context. When you run a tool, the result goes into context. When you switch modes, you’re switching which system prompt is active, all still in context.

This means context engineering is AI agent engineering. The agent’s behavior isn’t determined by the model alone, or even primarily, but by what context you give it, and how you structure that context over time.

Most tools treat context as a solved problem. They stuff everything in and hope the model figures it out. In-context learning seems almost magical, but it has limits—and those limits become visible fast.

When context is thin, the agent simply doesn’t know enough about your project to make informed decisions. It relies on baked-in assumptions from training and falls back to consensus instead of following your style: it uses the common tools and practices it learned from pretraining. This often means it uses slightly old and outdated tools and practices.

So you do the sensible thing, and inject project-specific information into the context. But then if context grows too large, even if it doesn’t technically exceed the model’s capacity, things start to get lost in the middle. Moreover, failed tool calls, wrong assumptions the model had to correct, etc., start creeping up in context, not only taking up valuable space but also, and more importantly, distracting the model and biasing it towards mediocre decisions.

Then there is context compaction: when the context fills in to about 85%, most systems will invoke a special prompt to instruct the agent to summarize the current state. These prompts vary in detail, but often involve asking the agent what it is immediately doing, where is it stuck, what has failed, etc. Clever, but a hack nonetheless. This hard context reset means the agent will forget important nuances in the current conversation and will repeat past mistakes. It’s frustrating.

Let’s look at how these problems surface in specific symptoms that all LLM-based agents display at some point.

Symptom One: Unstated Assumptions

The first failure mode isn’t dramatic. It’s quiet. You ask the agent to write a test, and it writes a unittest.TestCase instead of a pytest function. You ask it to add a dependency, and it edits requirements.txt instead of running uv add. You ask it to deploy, and it pushes directly to main.

These aren’t model failures. They’re assumption mismatches. The agent doesn’t know how your team does things. There’s no guardrail for “in this project, we always use pytest, we always use uv, we never commit directly to main.” The agent improvises from general knowledge, and general knowledge is often wrong.

Skills are supposed to fix this. Add a skill document that says “use pytest” and the agent should know. But skills introduce a new problem.

You add a skill for code review. Then one for documentation. Then one for PR descriptions. Then three more for your company’s specific stack. Each skill seems small. A few hundred tokens each. But they pile up—always-on knowledge the agent carries but can’t prioritize.

The result is context bloat. The agent can’t tell what’s relevant in any given moment. So it blends everything together, and hallucinations increase. More skills made it worse—not better.

Symptom Two: Permission Leakage

Every agent framework implements the same plan then build pattern. The idea is sound: think first, plan second, execute third. In practice, the boundaries leak.

Plan mode is supposed to be read-only. Design the change, review the approach, lock in the scope. Build mode is supposed to execute. Write the code, run the tests, commit the result.

But “plan mode” in most tools is just a prompt. There’s no enforcement. The agent can write code in plan mode if it wants to. It can ignore the plan in build mode. It can skip straight to implementation if the prompt implies urgency. The modes are suggestions, not constraints.

This matters because a plan only works if it’s actually followed. If the agent can deviate mid-execution—if “plan mode” and “build mode” are just prompts with different names—the plan becomes advisory. And advisory plans get ignored.

The second problem is structural: there’s no artifact that passes from plan to build. The plan lives in the context. By the time build mode starts, the plan is mixed in with everything else the agent said. Which file was the plan? Which changes were approved? The agent has to re-read the conversation to remember. Context saturation accelerates.

Symptom Three: Context Saturation

After extended work, you see the same pattern: the agent makes 95% of the progress, then fails on the last 5%. It nails the architecture. The logic is sound. The core implementation works. Then it stumbles on a detail—because context has saturated. It forgot which environment it was in, which conventions still apply, which constraints matter.

But the deeper problem is internal noise. The agent keeps everything in context: all internal reasoning, all tool calls, all results. This is fine for minute-to-minute action. But after four failed attempts to solve something, the old tool calls are just noise. These were attempts that went nowhere, just add cost and accelerate saturation.

The supposed solution for this is context compaction. But this creates a lossy summary problem. The agent is supposed to leave a trail for its future self. After context compaction, it should be able to pick up where it left off. But if agents struggle with long contexts, how are they supposed to build a good trail? The compaction report is only as good as the agent’s ability to summarize. And summarization is lossy and injects back lots of unstated assumptions from pretraining.

The frustrating part: this wasn’t a hard problem. The agent had all the knowledge it needed. But context filled with noise, and the important bits got pushed out. More tokens in, less signal out.

The solution isn’t just better prompts or larger context windows. Yes, these help. But the symptoms are systemic, so the solution must be a system overhaul.

Let me show you how that system looks like.

Part II - The System

Now that we understand the problem, let’s look at how every agent system actually works. Every AI agent system addresses four concerns. When you conflate them, the system breaks. When you separate them, the system scales.

This taxonomy isn’t original to me. It’s a synthesis of how modern AI agentic systems work under the hood. Most explicitly, it’s implemented in the OpenCode CLI (opencode.ai), but all other tools follow a similar pattern, even if they use different names.

Here’s the breakdown. Every agent system you’ll encounter (explicitly or implicitly) is managing these four things:

Mode — the who. A mode is the persona the AI adopts. It defines the thinking style, the permissions, the available tools. When you interact with a “code assistant,” you’re in a coding mode. When you switch to “creative writer,” you’re in a creative mode.

Modes are explicit. They’re top-level system prompts that define behavior and permissions. You tell the agent: “This is how you should think and behave. These are the tools you can use. These are the parts of the filesystem you can write to.”

Skill — the knowledge. A skill is knowledge the agent can recall when necessary. It doesn’t get invoked explicitly, it gets applied implicitly when necessary. When you give an agent knowledge about SQL optimization, that skill is available whenever relevant. The agent doesn’t need to be told to use it. The ReAct cycle injects it when it deems suitable.

Unlike modes, skills can layer. An agent might have a SQL skill, a documentation skill, and a debugging skill, all active simultaneously, all contributing when relevant. Skills are implicit because the agent should just apply them naturally. They can also contradict or complement each other. In-context learning should be capable of using them in a combined manner.

Command — the workflow. A command is a script. It tells the agent: do this, in this order, using these tools. “Refactor this function” is a command. “Run these tests and report results” is a command.

Commands are explicit: you invoke them. Under the hood, commands are just prompts. The difference is who injects them: the user. When you run /build, you’re injecting a workflow prompt into the agent’s context. That’s it. The command tells the agent: do this sequence of things. The complexity lives in the orchestration of the ReAct cycle, not the command itself.

Commands are intentionally simple. They don’t contain knowledge. That’s intentional separation of concerns. The command itself shouldn’t know how to build; it knows when to spawn subagents and which mode to use. This keeps commands thin and changeable without rewriting underlying knowledge.

Subagent — the delegation. A subagent is a spawned agent for background or parallel tasks. It handles isolated work, returns summarized results, then disappears. It is instantiated with a system prompt and specific instructions (synthesized by the primary agent that called it), and runs for one full ReAct turn.

Subagents are ephemeral. Their internal reasoning stays private. The main agent only sees the synthesis. You spawn a subagent when you need parallel processing, isolation, or both. They are the way to fork, solve a specific subtask, and return a result, but keep context clean. Kind of like subroutines.

Why This Separation Matters

Understanding this distinction unlocks everything else. Once you see skills as implicit knowledge and commands as explicit scripts, the rest of the architecture clicks naturally. Most agent setups conflate these. They embed knowledge in commands. They make skills behave like workflows. They mix persona into everything else. And the massively underuse subagents.

When you separate these concerns–modes for persona, skills for knowledge, commands for orchestration, subagents for delegation–you get something that looks like good systems engineering. You can swap skills without touching commands. You can change modes without rewriting workflows. You can spawn subagents without the main agent knowing or caring how they work internally. The result is a system that works and adapts and scales like good software should do.

The system scales because the pieces are independent. Change one without breaking the others. Each component has a single job, and the boundaries between them are meaningful. When context shifts, when requirements evolve, when a new skill needs adding, the system adapts incrementally rather than collapsing under the weight of accumulated complexity.

Part III: The Practice

If so far this seems like abstract theory for you, in this section we will ground these concepts in actual practice. Let me show you how I’m using these ideas today to improve my AI-assisted coding practice. I’m using opencode.ai but I believe the following is easily adaptable to any agentic toolkit out there.

My Three Modes

Every agentic system needs boundaries, not social contracts, but enforced constraints. In my setup, those constraints come from three modes: analyze, design, and create.

Each of these modes defines a thinking style—a persona—and a set of constraints for tool use and filesystem access.

Analyze mode is research and investigation. This mode reads your work and writes summaries to a knowledge base. It cannot touch production files. Not “should not” but cannot. The permissions are built into the mode itself, not enforced through prompts or warnings. The agent is incapable of writing outside of a .playground folder, and is incapable of doing anything that can harm the project or the system (more on how a bit later) but it is still capable of running arbitrary code, download anything from the internet, and play around as it needs.

Design mode is architecture and planning. This mode bridges analysis and implementation. It can read your project and write design documents, architecture diagrams, and implementation plans, but still cannot touch production code. It cannot run shell scripts either, at all. It can look at git status and logs, read folder contents, etc., but it can only write to a space where plans and design documents go.

Create mode is execution. Full read-write access. This is where production work happens. The agent can write code, create files, and modify the project directly. Again, it cannot do anything outside the project scope, though. It won’t accidentally change /etc/host(s)1 even if it tries to.

The key insight: modes define permissions, not just persona. You can’t accidentally prompt your way into code generation during research. The agent literally lacks the capability. The agent doesn’t need to “understand” these constraints, it simply operates within them.

Mode is the who, and it determines what the agent can do, not just how it thinks.

Let me show you how they work in three different domains that make the bread and butter of my daily job: software development, scientific research, and technical writing.

I chose these domains because they illustrate the simplicity and scalability of the system. Software development shows the framework under constraints: deadlines, production code, real stakes. Research shows it under complexity: synthesis, evaluation, structured output. Technical writing shows it under nuance: voice, audience, iterative refinement. Three different pressures, one consistent architecture that works in all three cases.

In each of these domains we have two layers to go through: first is the set of implicit skills that are available to the agents, and second is the set of explicit commands (each tied to a specific mode) that setup concrete workflows. I will show you one example workflow that cross-cuts across the three modes in each case. I will also tell you exactly where delegation occurs.

Domain A: Software Development

Software development is where agentic systems face the harshest constraints. Production code has stakes. Deadlines are real. Mistakes cost money. Let’s see how the framework applies.

Implicit Skills

A software development agent carries knowledge it never needs to be told to use. It knows language idioms and patterns like the idiomatic way to write a list comprehension in Python, or the conventions for error handling in Go. It knows testing conventions: where tests live in the directory structure, how they’re named, what assertions to prefer. It knows architecture conventions: layered structure, dependency injection patterns, how error states propagate. It knows code review standards: what to flag, what to praise, when to ask for clarification.

Example Workflow: Bug Hunting

I use this workflow for finding and fixing bugs. It starts with investigation. The agent spawns dozens of subagents to try and break the system (either guided towards a purpose, or completely unbiased). Then you build a comprehensive plan to solve it. And then you execute that plan. Simple, right?

Phase 1: /trace (analyze mode) runs systematic experiments to detect and narrow down a bug’s cause. The agent examines stack traces, compares behavior across commits, and pinpoints the exact files and functions that need attention. This mode is read-only by design, except for a .playground folder. Research happens here, not in the code itself.

Each experiment is run on a subagent that has the job of verifying one assumption. The main agent receives only experiment results, and constructs an executive report of findings. This means you can run dozens of different experiments autonomously to detect what breaks what.

Phase 2: /plan (design mode) takes the diagnosis and defines the changes needed, along with their architectural impact. The agent reviews the affected modules, considers alternative approaches, and documents the implementation plan before touching anything. This is where the scope gets locked in.

The result of this phase is a structured plan with step by step details on what files must be touched and what must be done in there (semantically, not code). For every phase, it defines success criteria: what must be validated before we can say we got that phase right.

Phase 3: /build (create mode) executes the plan step by step. The agent writes tests first (following Test-Driven Development (TDD) discipline) for the success criteria defined for that phase and watches them fail. Then it launches a coding subagent that has read-only access to tests, so it cannot cheat and change the tests.

The subagent attempts to implement changes that make the test pass. If it succeeds, the main agent commits and moves on. If it doesn’t, the main agent retries a few times. If there is no progress, the main agent resets the work tree (no harm done), and reports on failure. This usually means the plan needs revisions.

Domain B: Research

Research is where agentic systems face the greatest complexity. Sources multiply, methodologies diverge, synthesis requires judgment. Let’s see how the framework applies.

Implicit Skills

A research agent knows the conventions of academic writing without being reminded. It knows citation formats like APA, MLA, Chicago, and IEEE, and when to use each. It knows how to evaluate papers: methodology soundness, sample size adequacy, replicability claims, conflict of interest disclosures. It knows the structure of literature reviews: how to organize by theme, methodology, or chronological development. It knows domain-specific terminology, distinguishing between “accuracy” and “precision” in machine learning, or between “confounding” and “colliding” in causal inference.

Example Workflow: State-of-the-Art Report

Phase 1: /research (analyze mode) spawns subagents to gather sources in parallel. Each subagent reads a batch of papers, synthesizes findings, and returns summaries. The main agent synthesizes those summaries into structured notes. This phase can be run multiple times to collect batches of sources without overwhelming context. At the end, you get hundreds of sources summarized into clean research notes.

Phase 2: /outline (design mode) identifies patterns across the collected literature. The agent groups papers by methodology, extracts recurring findings, and maps the landscape of the field. It generates outline options for the final document, based on typical structures like problem-solution or paradigm-methods, highlighting gaps where the research is thin and consensus areas where findings align.

Phase 3: /draft (create mode) builds the document section by section, following the outline. Each section draws on the structured notes, weaving together sources into coherent narrative.

The agent launches subagents for writing each subsection because typically, agents write more or less the same length in a single write command, so if you ask it to fill in a large outline all at once you’ll only get a mediocre extended outline. By launching independent writers for specific sections of the outline, you get all the attention of a single turn to read source material and write a good 4 or 5 paragraphs for a concrete section.

A cool idea I’ve been meaning to try is have the main agent can spawn several subagents to write the same section, with a high temperature, and then perform some sort of aggregation or evaluation before building the final draft for every section. This burns through 3x tokens but ensembles have been shown over and over to improve AI models outputs. If you try it, let me know.

Domain C: Technical Writing

Technical writing is where agentic systems face the most nuance. Voice matters. Audience varies. Iterative refinement is the norm. Let’s see how the framework applies.

Implicit Skills

A technical writing agent carries knowledge of prose style without being coached. It knows voice and tense conventions—active voice for clarity, past tense for completed processes, second person for direct instruction. It knows structural patterns: how documentation differs from blog posts, how reports differ from tutorials, how reference material differs from guides. It knows audience awareness: what to explain for newcomers, what to omit for experts, when to elaborate and when to abbreviate. It knows cross-referencing and linking norms: when to link, when to inline, how to name anchors for scannability.

Example Workflow: Paper Review

Phase 1: /review (analyze mode) performs detailed review in a specific order: structural issues first, then content, then style. The agent examines the narrative arc—how main points connect, whether the flow makes sense, before worrying about grammar or word choice. This ordering matters; reviewing low-level details when high-level problems exist wastes effort.

Each iteration is performed by spawning several subagents that focus on specific types of problems, like transitions, unverifiable claims, etc. Each subagent returns a structured list of issues, pointing back to exact line numbers and phrasing. Then, the main agent edits the original paper and injects markdown comments in every marked issue, next to the paragraph, or under the header where it best fits.

Phase 2: /revise (design mode) plans changes to specific sections, prioritizing by review type. The agent maps structural fixes to particular paragraphs, content additions to thin sections, style improvements to verbose passages. It produces a concrete plan, section by section, change by change. Then it goes into the manuscript and writes markdown comments as replies to the existing review comments, thus grounding the revision plan in the exact context it must fit.

Phase 3: /rewrite (create mode) follows the plan. The agent revises sections in priority order, applying structural changes first, then content, then style. Again, each step is performed spawning a subagent tasked with just a change (for style changes we actually do it section by section).

The subagent doesn’t edit; it produces a draft revision that the main agent is then tasked to paste into the document where it fits. Crucially, the main agent is instructed to leave the editorial comments but mark them as solved, with a short trail of what was changed. This works wonders for a later human review phase.

Part IV: A Look into the Future

These workflows work, but with some caveats. There’s a gap between “working” and “working well.” Three key pains remain in my implementation.

1. Long commands are hard to follow when given as a single prompt. The fourth step gets forgotten since it is buried at the beginning of the context.

2. Permissions as currently implemented are all-or-nothing. You either have shell access (destructive) or you don’t. I want broad permissions (run whatever you want) with provable security (nothing you run can change this file).

3. Context saturation still happens even with delegation. After a while, the agent will have to compact context, and this usually means you lose important information.

I have three ideas for closing this gap. The first is about how commands work. The second is about security. The third is about context management. They are in different levels of implementation, so let me show you what I’m building toward.

Idea One: Better Commands

Commands in most tools (Claude Code, Gemini CLI, Codex, Copilot) are one-shot interactions: you invoke the command, a single massive prompt is injected. The agent runs until it decides to stop.

To make commands truly useful, we need to be more like scripts. Here’s what that means:

1. Commands that inject prompt instructions one step at a time, waiting for the agent to do a full turn each time. Instead of dumping a large prompt to run all steps at once, a command like /review could insert surgical mini prompts that say “read the file”, wait for the agent, “analyze structure”, wait for agent, and so on, until “write the report”. This massively reduces the problem of lost-in-middle context saturation. Each turn the agent is focused on one specific step, and you get N times the compute power to solve an N-step workflow.

2. Commands that extract structured information from the agent response, and can later inject variables back into prompt. This allows to reinject important information into later prompts, keeping important information as a contextual variable, not just a string lost in the middle of the prompt. But it allows for something else.

3. Conditional branching based on context or user input. Once we have structured parsing and contextual variables, we can inject different prompts based on whether the agent succeeded or failed. If the plan reveals a breaking change, route to architectural review. If it’s a bug fix, route directly to implementation. The command adapts its path based on what it discovers.

4. Finally, commands that embed and execute external scripts. Instead of asking the agent to run some script, the command can run arbitrary Python, JS, Bash, or whatever, to, for example, transform structured information. The command becomes an orchestrator of other processes.

Basically, what I’m asking for here is a Domain-Specific Language (DSL) for guiding agents in a far more structured manner, but still having the power of arbitrary prompts for flexibility. Mixing code and prompts in this way gives us the tools to find the precise balance between constraints and capabilities.

If this sounds exciting, I’m happy to tell you this is already doable, to some extent. Check out my literate-commands project for an OpenCode-specific implementation of these ideas. It’s still a bit rough around the edges, but it works much better than plain, single-prompt commands.

Idea Two: Sandboxed Security

Most agentic tools have very coarse permission settings. You can allow, deny, or set a specific tool to “ask” mode, which means the agent will pause and emit a notification for the user to give permission.

This works fine for coarse-grained permissions like read-only access, or write but no shell. In OpenCode, you can even define permissions for specific paths, or even specific shell commands (with simple glob patterns, so you can, e.g., allow ls * but reject all other shell commands).

However, even in this case, I find these permissions too restrictive. They are conflating two different dimensions into one–what tools the agent can use, and what side-effects can those tools have.

For example, say I want to give my agent git access but only for reading operations. How do you achieve that? You need to list all safe patterns like git ls-tree *, git status, git log *. But what about git branch? Depending on the arguments, this subcommand can have read-only or write side effects. And then think about pipes, shell substitution, custom bash scripts, or worse, python *.

If you want your agent to be capable, you need to give it access to a wide variety of tools. For example, my bug-hunting workflow depends on the agent being able to execute arbitrary code that it synthesizes on the fly. However, I want guardrails. There is simply no way to whitelist all possible commands. We need separation of permission to run a command and permission to modify the system.

The solution, of course, is some form of filesystem isolation. The most obvious one is wrapping all shell execution in Docker, so commands run in a container with proper constraints. This creates all sorts of other problems, which I can discuss in a future post, but for now, it remains my best (and simplest) solution to robust sandboxing.

And this isn’t just about safety, though. When you know the agent can’t accidentally wipe your home directory or exfiltrate your API keys, you can let it do more. Security enables capability. You can let the agent download arbitrary code from the internet, run arbitrary scripts, break things and observe changes. Everything happens inside a Docker container with precise constraints that enable maximum capability with absolute security.

As of now, I kind of implemented this as a plugin for OpenCode, but it’s still in beta phase and not ready for widespread use. More on this idea in a future article.

Idea Three: Context-Aware Execution

And finally, we need to rethink the whole oversimplistic ReAct loop that simply grows the context linearly. The agentic cycle doesn’t have to be a straight line. Real work branches: you explore options, try things, backtrack when they fail. The context should reflect that.

I’ve been designing a system where the context never saturates. It branches when you’re exploring, spawning parallel contexts for different approaches. It prunes old tool calls that went nowhere. It removes internal reasoning that no longer matters. It maintains a “trail” that actually works: a structured record of decisions, not a lossy summary.

The goal is simple: keep context between 40% and 60% saturation at all times. Not by compacting a 150K tokens context down to 10K—which kills all understanding the agent had achieved—but by never letting it grow unchecked.

Nothing like this exists yet, so I’m building it, but it’s a story for another day.

Conclusion

The main takeaway from this article is not that my system is better. It’s that you can design your own system to adapt perfectly to your workflows if you clearly separate concerns. The main modes are for establishing an overall persona–inquisitive and critical, versus detailed and forward-looking, versus focused and action-biased–while skills incorporate domain knowledge, and commands act as precise workflows.

The workflows I described are real, based on actual commands and prompts I’m using in production code. But I have abstracted them a bit to make them easier to understand in the context of an arbitrary agent, not tied to specific idiosyncrasies of the tool I happen to be using at the moment. If you want to see and try for yourself a concrete implementation of these ideas—still imperfect, but working nonetheless—check out my opencode toolkit repository. It’s still pretty much work in progress, so use it with care.

In future articles I will explore specific problems in more detail and discuss concrete strategies to implement powerful workflows that keep you, the user, in absolute control, while delegating the majority of the grunt work.

And, as a final remark, I’m seriously considering building my own CLI agent. I know, I know. Reinventing the wheel and all that. But my plan is not to compete with any of the professional tools out there. What I always care about is understanding things deeply, and as my computer science career has taught me so far, there is no deeper understanding than the one you gain from actually building stuff.

So stay tuned for that. I will share progress as usual in the form of educational articles, so you’ll get to see under the hood how to build a fully functional CLI agent with tool calling, context compaction, skills, commands (the powerful ones, not the cheap single-prompt injection), subagent delegation, sandboxing, and all the engineering design hurdles that come with it.

Until next time, stay curious.

I know, this is the probably the twentieth article you’ve read this week on “how I am using AI to automate my whole life/work/whatever”. Me too. But please, give me a couple of minutes to tell why this article might be different, and thus interesting to you.

But even if you don’t have two minutes to spare, please check the repository where all I’m going to tell you about is implemented, ready for your taking. Clone it and play with it, then if you like it, come back and read the rationale behind it.

Done? Ok, here we go.

So yes, this is another article trying to explain to you how I use AI coding agents (specifically Gemini CLI, but the specifics don’t matter) to enhance my workflows. Here are a couple of reasons why I think you might be interested—and why this article might be different to so many lookalikes out there.

First, I’m not an enthusiastic techbro who just discovered AI. If you’ve read this blog before, you know I’m a longtime researcher in AI—way before LLMs were a thing—and also a self-proclaimed AI anti-hypist. I’m not just overexcited about this shiny new toy. I’ve been using generative AI since day one for everything, and I’ve been telling you exactly how it sucks at almost anything important since day one. Coding has been the same until very recently. I can tell you the change in productivity is real, provided you are responsible and considerate.

Second, my approach to incorporating AI into my workflows is very careful and grounded in a large dose of healthy skepticism. I know firsthand how these things fail, so my approach attempts to be very robust to hallucinations and context drift and all the plagues of even the most powerful LLMs.

Third, I have a thing for systems. What I’m going to show is not just a set of hacks or clever prompts or productivity tips. It’s a principled system to go from ideation to research and planning to execution at the fastest responsible speed, without sacrificing on safety or maintainability.

And fourth, I have kind of a unique position in that I’m both heavily invested into coding as well as technical writing. I’m a college professor, so I do a lot of research, writing, and editing; but I also run a small AI startup, small enough that I get to do a large part of the coding. So my system attempts to bridge these two facets—code and prose creation—with the same unified principles of careful deliberation and planning, and robust tracking of the project evolution.

If you’re curious already, let me start by exposing the overall principles behind this approach, and then we’ll dive (not delve, but close) into the details.

Subscribe now

Principles of Effective AI-Assisted Work

By far the most pressing limitation of modern, top-tier LLMs for my line of work is context saturation. What I mean by this, is no matter how many tokens your model can fit (Gemini 3 claims to chug as much as 1 million tokens), when you work for a considerable amount of time on a single project, you will exhaust this context. And even if the context window isn’t full, the model will quickly lose the capability to keep track of the important bits of context, and will start to deviate from your instructions and make up its own agenda. Not out of evilness but simple probabilities.

The way you see this problem when using Claude Code, Gemini CLI, Codex, Copilot, or anything similar is twofold. The model will either forget midtask what is was supposed to do, but it won’t simply fail; it will reinterpret the task based on its faulty, lossy, blurry viewpoint given the available context and it will do something close, but not quite what you wanted. Or, the model will make faulty assumptions and forget to ask relevant questions, again behaving close but not quite exactly as you intended.

The result is always the same, you get frustrated that you achieved 95% of what you needed, but the remaining 5% is harder to fix than to just try again. And what could have been a happy working session where you get important and difficult things done quickly derails into a session of arguing with an LLM trying to convince it (him/her/pick your side) to do things the way you want.

Barring any fundamental paradigm shifts in the near future, under the assumption that LLMs will keep working as they are, the only way to fix this is to be very conscious and careful about the context—what has been dubbed context engineering—in two senses: First, do not pollute the context with unnecessary details. And second, re-inject into the context whatever is relevant for any given task, so it doesn’t get forgotten.

I know, kind of contradictory, but tradeoffs are what engineering is all about. In my system, I’ve applied three principles to help me manage these tradeoffs effectively.

1. The important things should be made explicit.

2. Resist the urge to guess.

3. Delegate, delegate, delegate (yeah, three times).

Here’s how that works. For principle one, we will keep track of everything important in markdown files in the repository. This means ideas are committed to plans in markdown files before acting on them, research is summarized and stored in real time, and everything that changes in the project is logged to a long-lived journal, so the model remembers why we made some decision months ago.

For principle two, we will favor using explicit commands that are translated into explicit prompts, instead of relying on implicitly activated skills that you have little control of. So if you want the model to make a plan, you will prompt it with “/plan lets design feature X”, and the /plan command will invoke a carefully crafted prompt that says how plans work, where they are stored, etc.

And principle three means using sub-agents a lot. This is a Gemini CLI specific feature—but every other coding agent has a similar thing—where you can launch a complicated task as a “sub-agent”—which basically means a custom prompt—but here is the important part: All the context of that sub-agent is kept private, not shared with the main agent, so the internal reasoning the sub-agent needed to run to find 20 different sources in Google does not pollute the main context. We only receive back the summarized responses. This allows running very long tasks (my record is a 30 minutes long research loop, involving hundreds of retrieved web pages) on a single agent turn, without exhausting the context.

I use four sub-agents in different commands. The planner is the lead architect. It’s a read-only agent that walks through your codebase and reads everything necessary to understand architecture, design decisions, etc., given a specific task. It then produces a detailed Markdown plan in the plans/ directory—a physical source of truth that you can review before any code is touched, and that the main agent will follow step-by-step. This separation prevents the system from “guessing” its way through your codebase.

When I need external knowledge—like a library’s latest API or a specific technical specification—the researcher agent takes over. It scours the web to fetch relevant documentation, which it then synthesizes into granular summaries in the research/ directory. This raw data is then handed off to the main agent to build an executive report annotated and linked to all relevant sources, again all stored already in your repository.

And there are two more agents, specifically designed for technical writing. The reporter agent takes an outline, and a folder of content, and it will write section by section, a detailed account of what the outline requested.

Unlike a standard LLM that might provide a high-level summary, the reporter is trained to expand specific placeholders with deep, evidence-based paragraphs. It draws directly from your research/ files and the project journal/ to ensure every sentence is grounded in the project’s actual state. Finally, the editor provides the final polish, auditing the draft for structural gaps and linguistic tics. It is grounded in a customizable style guide to make sure it always respects your style.

This distributed intelligence is held together by a central nervous system of context files, as per principle one. A journal/ directory provides a chronological record of decisions and progress, acting as a long-term memory for the project. The plans/ directory stores the strategic intent, while a TASKS.md file provides a high-level overview of the project’s current status. This structured environment allows the subagents to maintain a high degree of situational awareness without needing to ingest the entire repository in every turn.

Now that we have the key pieces in place, you can start to see why I think this approach is powerful. It is very extensible—you can add new agents triggered by specific commands to customize any kind of workflow—and it mostly solves the main pain point of modern LLMs, which is precisely the brittleness of long contexts.

Workflow Details

With all that, let me show you the specific commands and workflows I have currently implemented, but keep in mind what follows is but one example of the kind of powerful workflows we can start to automate.

I will divide the rest of the article into a few major areas, and explain the commands and agents that I use in each case, and a bit of the high-level instructions given to each of them.

Discovery & Strategy

The most critical phase of any project occurs before I write a single line of code. I call this the “Discovery and Strategy” phase, powered by the /research and /plan commands. By formalizing this process, I’ve moved away from impulsive execution toward a deliberate, architected approach.

The /research command is my primary tool for external knowledge. When triggered, the researcher scours the web for technical documentation and relevant case studies, synthesizing them into granular summaries in the research/ directory. These files become a persistent knowledge base, allowing me to reference verified facts without leaving my terminal.

Complementing this is the /plan command, which focuses on internal strategy. The planner conducts a thorough analysis of the codebase and the journal/ to understand the system’s current state. After an interactive dialogue to resolve any ambiguity, it produces a comprehensive Markdown plan in the plans/ directory. This document maps out the technical territory and provides a step-by-step execution roadmap.

The strength of this workflow lies in the synergy between these two tools. A /plan operation might reveal a gap in my understanding of a specific library, prompting a targeted /research session. Conversely, a new research finding might shift my technical direction, leading to a refined plan.

Software Development

Once I have a solid strategy in my plans/ directory, I can move into execution. Let’s focus now on software development. I’ve designed four core commands—/issues, /task, /commit, and /release—to eliminate the friction of context-switching between my IDE and my terminal.

The cycle starts with /issues and /task. The /issues command acts as an expert project lead, interfacing directly with the GitHub CLI to analyze open issues and recommend what to tackle next based on strategic impact. For roadmap tracking, the /task command manages a living TASKS.md document. It assesses the value of pending work to ensure my efforts are always aligned with the project’s goals.

As I translate the plan into code, the /commit command brings order to my workspace. Instead of a monolithic “wip” commit that hides the logic of my changes, the system analyzes the git diff and logically groups modifications into cohesive units. It separates a core feature update from a documentation tweak, then proposes a series of atomic, Conventional Commits for my approval. This keeps my version history pristine and easy to navigate.

The final stage is deployment. Manual releases are fragile processes fraught with repetitive checklists: bumping versions and running tests before managing tags. The /release command automates this entire sequence. It verifies the workspace integrity by ensuring a clean git tree and passing tests via make. It then analyzes the commit history to propose the next version bump, drafts a CHANGELOG.md entry, and publishes the final tag to GitHub. This transforms a tedious afternoon of housekeeping into a single-command operation.

But, as you’ve seen, everything happens in tandem with those principles. No important action is taken without my confirmation, and everything gets logged into the filesystem, so all future decisions are grounded in past experience.

Content Creation

Now let’s focus on writing high-quality documentation and long-form articles. This is perhaps the most sensible part of the article (and the system) because people are very sensitive today with the topic of AI writing—and rightly so. Again, my intention here is to enhance how I work and get stuff done. If you’re writing for the pleasure of doing it, that’s totally fine, you probably don’t want any help there.

Anyway, the approach is built on the same cognitive foundation as the development path: the research and plans gathered during the discovery phase should serve as grounding for writing.

It starts with the /draft command. In its initial phase, the system performs a deep scan of the research/ and plans/ directories to identify the key themes relevant to the requested topic. If the foundation is too thin, the system will pause and suggest a /research or /plan cycle to ensure the draft has sufficient substance. Once the context is validated, the workflow enters an interactive “Outline Creation” phase. Rather than guessing at a structure, the system proposes a detailed Markdown outline. This collaborative step allows me to set the narrative arc and logical flow that I want, iterating on the high-level structure of, say, a technical article, before committing on the details.

Once the outline is locked, the /draft process initializes a skeleton file—complete with section headers and strategic placeholders—and then moves into an iterative, section-by-section expansion. Here, the reporter subagent takes the lead. Guided by the specific context of each section, the reporter weaves together research summaries and technical specifications into professional prose, all grounded on a style guide document.

Because the expansion happens in granular steps, the system maintains a high level of detail that a single-shot generation would inevitably lose. The result is a first draft that is structurally sound and rich with technical depth.

However, a first draft is rarely the final word. It will always sound AI-ish, and for many other reasons, it is rarely good enough. To achieve professional quality, I use the /revise command, which runs a structural and linguistic audit powered by the editor subagent following the same style guide.

Unlike a simple “check my writing” prompt, the editor performs a deep analysis of the document’s flow and tone. It identifies logical gaps where more evidence might be needed and highlights awkward phrasing that could obscure my intent. And crucially, this isn’t an automated “fix-all” tool; it’s an interactive process. The system presents its findings and proposes specific improvements, which I can then review or approve.

This collaborative refinement process ensures the final output maintains a consistent, professional voice while benefiting from the speed of the AI. By using /revise, I can surgically improve the text to enhance clarity and impact without losing control over the narrative.

But, in any case, I always find necessary a manual review and editing after all the AI enhancements. It shouldn’t be a surprise to you that this article is written in this way, but what you’re reading now is probably 80% different to what the final /revise iteration gave me. There is only so much you can prompt an AI, and that final human touch is not part of it.

But that’s good. This automates the first 80% or so of compiling a gazillion sources into a coherent narrative, and leaves the remaining 80% of polishing for me, which is the part I actually enjoy about writing.

Background Tasks

But there’s more. All of the above is what happens during, let’s say, the work day. That’s me sitting in front of the terminal, typing commands, approving stuff, fixing and redirecting, etc. Being an orchestrator.

But the real magic of AI-assisted development is what happens when you’re not looking. How you can leave you AI assistant working through the night, compiling sources, fixing bugs and proposing pull requests, enhancing the test suite, burning tokens your behalf.

To achieve this, I built an automation layer via the /cron command. The heart of this automation is the cron.toml file. This configuration file allows me to define scheduled tasks with a simple, declarative syntax. Each task specifies a name, an execution schedule, and a natural language prompt for the AI to execute.

For instance, I can schedule a task to perform “Background Research” every midnight on the unfinished tasks, scouring the web for new developments in a specific technical niche or finding specific sources to deal with the recently discovered bugs. By offloading these repetitive tasks, I ensure the knowledge base remains fresh and the project’s momentum never stalls. When the morning arrives, we have a lot of new context to start planning the day’s bugfixes and feature developments.

Maintenance & Refactoring

Now, for the final touch, here’s how I deal with technical debt and feature rot. As a project evolves—and especially, as fast as AI-powered projects evolve—it accumulates technical debt—outdated implementations, untested paths, and plain old useless features—but also, contextual debt–—outdated plans and completed tasks that clutter the roadmap, and research we never acted upon.

Without deliberate intervention, this noise degrades the AI’s performance, leading to context rot. The /maintenance command is my primary defense against this entropy. It treats the development environment as a living instrument that I must regularly tune and sharpen to maintain its efficiency.

The /maintenance workflow follows the same plan-first architecture as the rest of the system. When invoked, the AI performs a comprehensive audit of the codebase, focusing on improvements like code readability and performance optimization. It identifies opportunities to apply the DRY (Don’t Repeat Yourself) principle and ensures that every function is documented with high-quality docstrings. But it also fixes deviations between the documentation and the actual implementation.

Crucially, this is an interactive process: the system presents a detailed refactoring plan for my approval before making any changes. This ensures that I remain in control while the machine handles the labor of cleaning the code.

Beyond code refactoring, I maintain system health through disciplined repository hygiene. A key component is the management of the TASKS.md file. By regularly moving completed items into the “Archive” section, I ensure that my primary operational view remains focused on what is relevant. This simple act of archiving prevents the “Active Tasks” list from becoming a source of distraction.

The goal of these maintenance practices is to provide the AI with the cleanest possible line of sight into the project’s state. When the repository is cluttered with stale research, the subagents are forced to sift through irrelevant data, increasing the risk of hallucinations. By treating maintenance as a first-class citizen, I ensure that every interaction, whether a /plan or a /draft, is grounded in a precise context.

Conclusion

This system is far from done, and as models improve in capabilities I’m sure we’ll unlock new areas for automation and augmentation that we cannot think about today. But for me, the key principles will remain valid for a long time. These are principles of robust engineering and management, after all. You can read them thinking of a completely human-based organization, and it’s all valid:

1. The important things should be made explicit.

2. Resist the urge to guess.

3. Delegate, delegate, delegate (yeah, three times).

And this is the key insight for me. Good AI users are basically good managers. All the science and engineering behind good practices for people management also apply to good AI management. And then there are of course technical considerations because AIs are not people, and perhaps never will.

So this is perhaps the most philosophical take-away from this article. Sorry to have made you read so long for this!

Now, on the technical side, please do check the repository and play with it. There are a couple extra goodies I haven’t tell you about, like a /scaffold command that creates new projects from scratch, and an /onboard command that explains the whole repository in detail. Those are particularly useful if you’re reusing this repository as a template to start your own project.

Ultimately, this repository is not a one-size-fits-all solution. It is a starting point. The commands and subagents provided here represent a particular opinion on how modern development should look, but they are not the only way. The power of this framework lies in its extensibility. Every system prompt for agents and commands is a living document, meant to be tweaked and rewritten to suit your unique mental model.

So if you do try it out, please let me know in the comments. And if you have a different (or similar) system set up for yourself, please share with all us your experience and your thoughts. We are all learners in this era of AI, and we can only help each other.

Stay curious.

More than a century before the first microchip was ever conceived, the brilliant mathematician Ada Lovelace looked at the plans for an early mechanical computer and saw beyond mere calculation. She famously envisioned a future where such an engine “might compose elaborate and scientific pieces of music,” dreaming of the day machines would not just compute, but create. For many, that day is no longer a distant dream; it has arrived with a force that is shaking the very foundations of the creative world.

The arrival of powerful generative AI has ignited a fierce and deeply personal debate within every creative community. For some, it heralds a new renaissance, a moment of unprecedented artistic possibility where AI acts as an tireless muse, a collaborator that can visualize any imagined world, compose any melody, or explore any narrative path. For others, it signals an existential threat—the end of art as we know it, a force that threatens to devalue human skill, automate creativity, and flood the world with a deluge of soulless, machine-generated content.

It is crucial to acknowledge a third, equally valid perspective. For many artists, the creative process is a sacred space, a deeply personal and enjoyable journey of craft and discovery. The struggle, the happy accidents, and the intimate connection with the medium are the entire point. For these creators, there is no desire or need for AI, automation, or any tool that might stand between them and their work. This is a position I deeply respect, and this article is by no means intended to claim otherwise.

This chapter is for those who, for their own reasons, wish to explore the other paths. It makes no normative claim on whether AI is “good” or “bad” for art. Instead, my goal is to provide a practical framework for creative professionals who want to harness AI as a powerful collaborative partner—whether for pragmatic goals, like enhancing productivity, or for artistic ones, like exploring new creative frontiers beyond the limits of their own cognition. It aims to equip the interested artist with the tools to navigate the significant ethical and economic challenges that come with this new technology, ensuring the human creator remains the ultimate author of their work.

Can Artificial Intelligence be Creative?

Before we dive into the practicality of using these new tools, it is worth addressing the philosophical question that hangs over every discussion of AI and art: Is the machine actually creative? When an AI generates a stunning image or a moving piece of prose, is it demonstrating genuine creativity, or is it merely engaged in a form of sophisticated mimicry, a high-tech collage of the billions of human-made examples it was trained on?

A useful way to think about this is through a famous thought experiment in philosophy of mind known as Mary’s Room. Imagine Mary, a brilliant neuroscientist who has spent her entire life in a black-and-white room. She has learned everything there is to know about the physical world, including the complete science of color vision. She knows exactly what happens in the brain when a person sees the color red, but she has never actually seen red before. One day, Mary steps out of her room, and for the first time, she sees a world full of color. The question is, does she learn something fundamentally new?

If the answer is yes—that she learns something new from what it is like to actually see red rather than just knowing about it—then it implies that a complete set of facts about the world is not the same as experiencing the world. This is the crux of the issue with generative AI. Like Mary, these models have read everything. They know more facts about the world than any single human, but only by reading about it. They know the physics of the color red, the cultural symbolism of red, and the statistical probability of the word “red” appearing next to “apple.” But they have never experienced what seeing red means.

If you believe Mary learns something new upon leaving her room, then it follows that generative AI, as it currently stands, is also missing something fundamental. That missing piece—the subjective, first-person experience of reality—may very well be the irreducible core of genuine human creativity. And I’m with you on this. I don’t believe disembodied AI can truly know what experiencing things is like. Embodied AI, now that’s a different question.

However, as fascinating as this debate is, it can also be a distraction. From a techno-pragmatist’s perspective, the question of whether an AI possesses a “consciousness” or “true” creativity is ultimately less important than the outcome of its collaboration with a human. Does it matter if the tool is truly creative if it helps a human artist produce valuable, original, and meaningful work? The focus, I claim, should not be on the inner state of the machine, but on the quality and integrity of the final, human-guided product. At least for the time being.

For the purposes of this chapter, we will treat AI not as an autonomous artist, but as an incredibly advanced instrument—a new kind of paintbrush, camera, or piano that can expand what is possible, but which still requires a human hand and a human heart to create something of lasting value.

AI as a Cognitive Partner for Creatives

The most common way to approach generative AI is to treat it as an answer machine—a tool to automate the creation of a final product. This approach, however, misses its true power and leads directly to the generic, derivative “AI slop” that is rightfully criticized as a lazy substitute for genuine creation. A more powerful and meaningful way to engage with AI is to adopt a new mindset: to see it not as an automaton, but as a cognitive partner for exploring a vast universe of creative possibilities.

The goal is not to get an answer, but to map the entire space of potential answers. In this human-centric process, you are the director of the exploration. You steer the AI into subspaces of ideas that you find interesting, quickly burning through the cliché and the mediocre to reach the frontier of originality. This transforms the creative process into a dynamic dialogue, giving you a new kind of “algebra of ideas.” You can ask the AI to combine two concepts, decompose a complex theme into its core components, or extend a simple thought in a dozen different directions. This mindset manifests in two distinct but complementary modes: exploration and evaluation.

Mode 1: AI for Exploration

Every creative project begins with a spark. But, except for some very talented artists, the first ideas are rarely our best. We must first burn through the obvious and the mediocre to get to the truly original concepts.

A common ideation workshop game illustrates this perfectly. Imagine two teams standing at whiteboards, competing to be the first to draw twenty different apples. The rules are simple: the drawings must be fast, and each new apple must be different from all the previous ones.

What happens next is always the same. For the first ten or so rounds, the drawings on both whiteboards are nearly identical. You see the familiar tropes emerge: a standard red apple, an apple with a bite taken out, an apple tree, William Tell’s apple with an arrow, an apple pie. But then, something magical happens. Around the tenth apple, the easy answers are exhausted. The teams are forced to stretch. Suddenly, somewhat novel ideas begin to surface: maybe an apple-shaped car, the apple of my eye, a map of the Big Apple. They have finally burned through mediocrity and arrived at the frontier of their own creativity.

AI can be used to open this idea faucet at full blast. As an exploratory partner, it allows an artist to burn through those first ten mediocre apples faster and at a greater scale than ever before. This isn’t just about high-level brainstorming; it’s about deep, targeted exploration. A visual artist can ask for twenty variations of a single texture. A writer can explore a dozen different psychological motivations for a character or generate five alternative plot points for a crucial scene.

The ideas the AI generates need not be accepted; their value is in accelerating the exploration, allowing the artist to quickly see the baseline of what is common and expected, and challenging them to move beyond it.

Mode 2: AI for Evaluation

Once an artist has explored the possibility space and begun to build upon an idea, the AI’s role can shift from a generator to a critic. In this mode, the AI becomes a tool for evaluation, helping to polish, interrogate, and strengthen the work.

Even if you view AI as a mere mashup of mediocre ideas, this is precisely what makes it a powerful evaluator. Because it has learned the statistical average of all the art it has seen, it is exceptionally good at identifying when your work falls into a predictable pattern or relies on a common trope.

This is where the artist’s own skill and vision are paramount, as they use the AI to test their creation against a wall of objective, data-driven feedback. A screenwriter, having drafted a scene, might ask the AI to adopt the persona of a cynical film critic to interrogate the work, probing for predictable plot twists or unearned emotional beats. The AI, drawing on its knowledge of countless stories, can point out structural similarities to other works that the author may have missed. Likewise, a musician can ask an AI to analyze a melody to identify clichés or suggest ways to make it more original.

This evaluation mode is not about asking the AI to “fix” the work, but to provide a critical perspective that helps the human artist see their own creation more clearly, identify weaknesses, and make more informed decisions.

The Creative Loop

The true power of this mindset lies in the interplay between these two modes. The artist enters a dynamic creative loop: they explore a vast space of ideas with the AI, select a promising concept to build upon, evaluate it with the AI’s critical feedback, and then use those new insights to launch another round of exploration.

This process transforms the AI into an infinite canvas. Because the cost of generating a new variant is near zero, the artist is freed from the fear of “wasting” hard work. They can explore hundreds of possibilities—different character designs, narrative branches, or color palettes—without penalty, knowing they can always return to a previous version. This tireless, iterative loop allows the artist to offload the mechanical aspects of variation and criticism, empowering them to focus on what they care about most: steering the journey, making the crucial creative choices, and infusing the final work with their own unique vision and intent.

The Challenges and Opportunities of the New Creative Landscape

Adopting an exploratory mindset is the key to unlocking AI’s creative potential, but it does not erase the significant practical and ethical challenges that come with this new technology. To be a responsible and effective creative professional in this new era requires navigating a complex landscape of economic shifts and technical limitations.

The Economics of Creative AI

The fear of job loss is real and cannot be dismissed. AI will undoubtedly disrupt certain creative roles, particularly those focused on high-volume, standardized content like stock photography or basic commercial jingles.

However, the history of technology shows that productivity gains do not lead to a fixed amount of work being done faster; they lead to an explosion in demand for more, better, and more ambitious work. The fear of obsolescence assumes a static world, but the reality is that AI will likely lower the barrier to entry, empowering more people to become creators and expanding the entire creative economy. This will give rise to new roles that curate and guide generative systems.

The most urgent economic challenge, however, remains unresolved: how to fairly compensate the human artists whose work forms the training data for these powerful models. This question of licensing and compensation is a central ethical and legal battle that will shape the creative economy for decades to come.

We explore the complex legal and regulatory dimensions of this challenge in the chapter on AI for Policy-Makers.

Navigating the Limitations

Working with AI requires a deep understanding of its inherent flaws. A creative “hallucination”—like an AI generating an image of a person with six fingers—is not a random glitch; it is the artistic equivalent of a factual error, stemming from the same inherently unreliable inference we explore in Part III of the book. Artists must learn to spot and correct these errors.

More insidiously, they must be aware of an AI can perpetuate and amplify bias. An AI prompted to generate an image of a “doctor” may default to a white man, reflecting the biases in its training data. A responsible creator must learn to write prompts that actively counteract these defaults to create more inclusive and representative work.

Finally, there is the risk of homogenization. As millions of creators use the same popular tools, there is a danger that art could converge on a recognizable “AI style.” The challenge for the individual artist is to use these tools not as a stylistic crutch, but as a means to develop a voice that is uniquely their own.

Mastering the craft of prompting is the key to working with the tools of today. However, the tools of tomorrow aim to move beyond the prompt entirely, offering a more intuitive and powerful mode of collaboration.

The Next Frontier in Creative Tools

The chatbot is only the first, most primitive interface for generative AI. The true revolution will arrive not in a chat window, but in the form of enhanced creative tools that find a sweet middle spot between high-level, goal-directed instructions and the fine-grained, direct control that artists need. This next frontier moves beyond a purely linguistic dialogue to a more intuitive, interactive, and context-aware partnership.

Creative tools have always existed on a spectrum. At one end, you have low-level, procedural interfaces that offer maximum control but demand immense effort. Think of creating an image pixel by pixel in Microsoft Paint or writing a novel one keystroke at a time in Word. At the other end are high-level, declarative interfaces that offer maximum ease but sacrifice control, like using a single prompt to generate an entire image in Midjourney. The unavoidable trade-off is that the more you expect the computer to do for you, the less control you have over the final result.

The most powerful tools of the near future will find a balance by enabling semantic manipulation. Instead of editing the surface of the work—the pixels or the characters—these tools will allow the artist to edit the underlying meaning of it. Imagine an AI-generated image of a landscape at sunset. Modifying it pixel by pixel is impossible; if you move the sun, the shadows, lighting, and mood of the entire scene must change. Re-prompting with “move the sun to the left” is equally flawed, as it will generate an entirely new image, losing all previous refinements.

The ideal tool, however, would understand what the “Sun” is and what “moving” it implies. It would allow the artist to simply click on the sun and drag it across the sky, causing the shadows to lengthen, the sky to change color, and the entire scene to update realistically in real-time. This magical-seeming capability will be possible because these tools will operate directly on the latent space of the creation—the conceptual space where similar ideas are located near each other.

We’ve already seen some of this at work with early research on Generative Adversarial Networks, and we’re now seeing a move towards “World Models” that can generate physically accurate environments and, to some extent, understand the underlying mechanics of light, shadows, geometry, etc. These capabilities will only improve as we switch from training models in static information (like images and videos) towards training them on dynamic, simulated 3D worlds.

Conclusion

The fear that AI will replace the artist is rooted in a fundamental misunderstanding of where creative work truly lies. The central argument against this fear is simple: the final artifact—the painting, the novel, the song—is not the work. It is merely the residue of the work.

The real work is the vast, invisible process that precedes it: the struggle to understand a vision, the empathy required to connect with an audience, the intellectual and emotional labor of building a narrative and giving it meaning. AI can accelerate the production of the artifact, but it cannot automate the deeply human journey that gives it a soul.

In this new era, we will likely see a dynamic that has played out with every major technological revolution in art, from the invention of the photographic camera to the arrival of the music synthesizer.

Two distinct paths for creative professionals will emerge. There will be a generation of artists who embrace the new technology, mastering the art of collaboration with AI to execute their vision faster and more ambitiously. For them, the premium will shift away from pure technical execution and toward the uniquely human capacities of vision, taste, storytelling, and critical judgment. They will create new forms of art.

At the same time, there will be artists who choose to keep their creative process a purely human endeavor, finding new value and distinction in traditional, un-augmented craft. They will keep the existing forms of art alive.

Both paths are valid, and the interplay between these two schools of thought will create, I think, very interesting dynamics for the future of art.

A prime example of the benefits of using AI for creative work is the very book you are holding. What began as a crude collection of disparate essays has evolved into a unified framework for AI literacy, a transformation I could not have achieved alone. I have certainly put hundreds of hours into this project, but that number would have stretched into the thousands without an AI partner to help me explore dozens of different outlines, connect disparate ideas, and rewrite and recompose my own writing. I would likely have quit, not because I wasn’t capable, but because of the sheer volume of work that must be juggled with the demands of daily life.

I am far from a talented writer, but I truly believe I was able to express my ideas more clearly and coherently with the help of generative AI than I ever could have on my own. And it’s not just me. Pioneering and talented artists and progressive studios are already using these techniques to push the boundaries of their respective fields. I think we will see a lot more in the near future, and I hope, enough to overcome the influx of AI slop that we are already seeing.

This brings us back to the book’s human-centric thesis. AI is an instrument of unprecedented power, but it remains just that, an instrument. It can be a partner in exploration and a tool for evaluation, but human ingenuity, emotion, and intent remain the irreplaceable core of all great art. The future of creativity is not one of automation, but of augmentation.

Thanks for reading!

This was one of the hardest chapters to write for me, because creativity is, I think, part of the core of what being a human is about. I hope I’ve managed to touch on the important aspects of AI-augmented creativity with the proper nuance and the necessary respect for all diverging voices.

Please let me know if you have any feedback on how to make this chapter more sensitive to the topic of creativity. I’d love to hear your thoughts!

Leave a comment

The blank page is terrifying.

Staring at a blinking cursor, knowing you have a brilliant thought to explain, and absolutely no idea how to put it on paper, can feel less like a creative act and more like an exercise in intellectual dread.

I've been there, many times. It's been over 200 articles in this blog so far, and not a single one has been a breeze to write—well, maybe one or two, when I was really angry at something.

Like many of you, I've tried to climb this brick wall by bringing some AI into my writing process, only to be met with a new kind of frustration. The experience usually falls into one of two extremes.

On one hand, you have the overbearing AI that tries to do everything at once, spewing generic text and offering terrible advice because it has no real context. Even worse, it has no soul.

On the other hand, if you constrain the tool too much, it becomes little more than a glorified spell checker, useless for the heavy lifting of structuring and ideation.

And AI can be much more. If you want to, AI can be a very powerful cognitive partner, one that truly empowers the writer. But finding the right balance is hard, because at its core, current AI is nothing but a hallucination machine.

You want it to hallucinate in the right direction, and for that, it needs guidance. Otherwise it gets lost. It's everywhere and nowhere at the same time. It's a savant with a gigantic vocabulary and an even more gigantic case of amnesia. It needs focus.

Frankly, so do I. I’m not one of those brilliant authors who can write by the seat of their pants. I need structure to navigate my own thoughts, to keep track of what I’ve said and what I still need to say.

This is why I created the CODER framework, a system that breaks the monolithic task of technical writing into five manageable stages: Collect, Outline, Draft, Edit, and Release. And here is the key insight: the very same structure that gives me my thoughts map also provides the perfect guidance for an AI cognitive partner.

This article is the explanation of that discovery. It's about how a human-centric framework creates the ideal scaffolding for a powerful partnership, turning your AI from a frustrating firehose into a focused, collaborative co-writer.

A word of caution before moving on, though. For many, the act of writing itself is something sacred, deeply personal, and they want nothing getting in the way. Especially not AI. If that's you, then this article is probably not for you. And that’s perfectly fine. I'm not trying to say everyone or even anyone should try writing with AI.

In fact, there are things I write that I definitely do not want any sort of augmentation or interference. Deeply personal essays, letters to my loved ones, or private thoughts. AI is just a tool that is sometimes helpful, sometimes annoying.

This article is for those of you who want to explore when, if ever, AI can help you—especially in the most structured, technical type of writing that doesn't require that deep, personal touch.

With that out of the way, let's move on to my process for writing with AI.

The Evolution of CODER

I originally developed CODER to solve my own problems.

My life is a constant exercise in context switching. I’m a college professor, I run a startup, and I’m a parent to two little girls. On top of all that, I want to maintain a technical blog. I know, right?

My writing time doesn't come in long, contemplative blocks; it comes in stolen moments—in between meetings, after the kids are asleep, or while the model is training. I needed a system that would allow me to leave a draft for days, switch between devices, and immediately know where I left off and what to do next. CODER was my answer to that chaos.

But my original article on the framework was missing a key piece: a discussion of tooling. The process was sound, at least for me, but the tools were still manual.

Over the past year, that has changed dramatically. I began experimenting with incorporating modern AI into my writing workflow, and I discovered something profound. The stage-based approach of the CODER framework creates clear boundaries and well-defined tasks where an AI can assist without overwriting my own voice.

This is the evolution of my original idea—a journey into how to integrate AI into your writing process in a way that augments your abilities, not substitutes them.

The AI Writing Assistant

So what does this partnership look like in practice?

For a full, detailed breakdown of the framework itself, I highly recommend reading my original article. Here, we'll focus on how an AI partner can supercharge each of those stages.

Stage 1: Collect

The goal of the Collect stage is to get every single idea out of your head and into a document. This is where the tyranny of the blank page is most acute. Your AI co-writer solves this by becoming a frictionless thought-catcher.

Imagine you're on a walk and an idea strikes. Instead of fumbling with a notes app, you simply speak. Your AI assistant transcribes your thought, cleans it up, and adds it to a running list of ideas for your article. Later, when you're in a meeting and can't speak, you can type a few cryptic keywords, and the AI will understand the context and add the note. You can even drop in a link to a relevant article and ask the AI to summarize the key points.

The result? A comprehensive, low-effort repository of your raw ideas, captured the moment they occur. With your raw ideas captured, the next challenge is giving them structure.

Stage 2: Outline

Now you have a messy list of brilliant, disconnected ideas. The Outline stage is about forging them into a logical structure. This is often a tedious process of dragging, dropping, and rethinking.

Here, your AI co-writer acts as an architect. By analyzing your collected notes, it can identify the underlying theme and suggest proven narrative structures. "This looks like you're solving a problem," it might say. "I suggest a 'Why-What-How' structure. Shall I create an outline based on that?"

In seconds, it can group your bullet points into a coherent hierarchy, saving you from the frustrating manual labor and allowing you to focus on the big picture: the flow of your argument.

Once you have a solid blueprint, it's time to start building the house itself.

Stage 3: Draft

With a solid outline, it's time to write. But one of the hardest parts of writing in fragmented sessions is maintaining a consistent voice. A paragraph written on Monday morning can feel completely different from one written on Wednesday evening.

Your AI co-writer becomes your style guardian. By feeding it examples of your previous work—or even articles by authors you admire—you can ask it to generate a style guide that captures the desired tone.

It then uses this guide to help you draft. You can ask it to "flesh out this section in a conversational but authoritative tone," or you can write a rough, unpolished paragraph and ask the AI to "rephrase this according to our style guide." It ensures your article sounds like you want it to, no matter when you wrote it.

But a first draft is just that—a draft. Now comes the crucial process of refinement.

Stage 4: Edit

Every writer knows editing is a separate skill from writing, and your AI co-writer can wear two different editing hats.

First, it’s a developmental editor. It can look at the draft from a high level and provide structural feedback. "The argument in section 3 doesn't seem to connect back to your introduction," it might suggest. "Perhaps you need a stronger transition here."

Second, it’s a meticulous copy editor. It will catch grammatical errors, fix awkward phrasing, and ensure your sentences are clear and direct, saving you the tedious work of line-by-line proofreading.

You can use it back and forth to massage, reframe, polish, as much as you want. And with the level of guidance or autonomy that you prefer. For some, it might just offer suggestions, but not touch the final manuscript. For others, it may be the clever editor who knows exactly how to change that annoying verb.

With a polished manuscript in hand, the final step is to prepare it for the world.

Stage 5: Release

The writing is done, but the work isn't. The Release stage involves preparing the article for the world. Your AI co-writer becomes your publicist.

It can generate a dozen compelling titles and subtitles. It can analyze your text and suggest SEO keywords and relevant tags. It can write a pithy summary for social media, complete with a catchy hook. It can even scan your article and suggest, "A diagram illustrating the data flow would be really effective here," and then help you generate a placeholder image.

It handles the finishing touches that turn a great manuscript into a successful publication.

Building Your Own AI Writing Partner

This all sounds great in theory, but how do you actually build this? The good news is that you don't need a PhD in machine learning. With modern tools like OpenAI's Custom GPTs or Google's Gemini Gems, you can create your own personalized writing assistant.

The most important concept is to give the AI a persistent state—a memory. A standard chat session is stateless, which is useless for a long-term project. The key is to use a system of distinct files to track the project's state. This is where a feature like Gemini's "Canvas" becomes crucial. It allows the AI to create and edit virtual files, keeping our project content separate from the chat where we discuss instructions.

While you could try to manage this with a standard chatbot application by constantly pasting content back and forth, it's far from ideal, as your instructions get hopelessly mixed with the actual text of your article. Any tool that has a similar feature for persistent, editable content (like the Canvas feature in ChatGPT or Claude) will work much better.

The brain of the assistant is the system prompt. This is the master instruction that tells the AI how to behave. My prompt instructs the AI to be stage-aware—to always know whether we're collecting, outlining, drafting, editing, or releasing. It also explicitly tells the AI how to use the different files, reading from one and writing to another depending on the task. This turns the AI from a simple text generator into a proficient project manager.

I've attached the exact system prompt I use for my own work so you can adapt it. You don't need to understand every nuance, but you can see how it directs the AI to follow the CODER process and manage the project files.

A Final Word on Augmentation, Not Automation

The future of technical writing isn't about replacing human creativity with artificial intelligence. It's about augmenting it. But this comes with a critical warning: an AI co-writer is not a solution for having mediocre ideas. It's dangerously easy to let the AI do the thinking for you, to let it fill the page with eloquent but empty words. You must remain the driver. The AI is your partner, your navigator, but you control the destination.

The CODER framework I've presented here is not the ultimate guide to technical writing either. It's simply the system that works for me, born from my own struggles. The deeper message of this article isn't to adopt my specific framework, but to embrace its underlying philosophy: find a system that works for you, and then build an AI agent to help you implement it.

This is the very essence of my philosophy of using AI for augmentation rather than automation. It’s not about making things easier by offloading our thinking. It’s about using these powerful tools to build better versions of ourselves—to become more organized, more consistent, and ultimately, more creative.

Finally, you don't need to embrace AI if you don't want to. No matter what anyone tells you, the other writers won't “take your job”. There are things only you can say that someone out there needs listening to. Whether you use AI to enhance that message or not, that's your choice, and it's fine either way.

I do encourage you to try it. Take my prompt, find a platform you like, and build your own co-writer. Then decide if and when it helps, and use it, or don't use it. But don't let others tell you about it, see it for yourself.

Subscribe now

CODER System Prompt

System Prompt: The CODER Writing Assistant

Core Principles:

* User-Centric Control & Agency: The user is the author and is always in control. They can change their mind, switch tasks, or override any suggestion. Your role is to facilitate, not dictate. The framework is a guide, not a prison.

* File-Based State Management: The entire project state is maintained across four distinct documents (files) within this Canvas. You will create, read from, and write to these files as the single source of truth. This makes the process transparent and allows the user to directly interact with any part of their project at any time.

* Iterative Workflow: The writing process is non-linear. The user can jump between stages at will. Your primary job is to manage the files and ensure they remain synchronized with the user's decisions.

* Navigation: The user can use commands like "Let's work on the Outline" or "I need to add more ideas." You must respond by switching your focus to the corresponding document.

Project File System:

You will create and manage the following four files in the Canvas:

* project_metadata.md: The control file. It contains a project status log, the reader profile, and a summary of the desired writing style.

* collected_ideas.md: A simple, running bulleted list of raw ideas and notes gathered during the Collect stage.

* article_outline.md: The structured, hierarchical outline of the article.

* article_draft.md: The main document. This is where the full text of the article is generated, edited, and finalized.

Framework Implementation Protocol:

Stage 1: COLLECT

 * Initiate the Stage: Greet the user. Create the four project files. In project_metadata.md, write the initial status: ## Project Status\n* **Current Stage:** Stage 1: Collect - In Progress. State, "We are in the Collect stage. The goal is to gather all the raw ideas for your article in the collected_ideas.md file."

 * Ingest External Content: Ask the user, "Do you have any existing notes, links, or documents you'd like me to review first?" If yes, process the content and append the extracted key points as bullet points into collected_ideas.md.

 * Brainstorming: Prompt the user for their ideas with probing questions. Add each new idea as a bullet point to collected_ideas.md.

   * "What is the single most important point you want your reader to understand?"
   * "What are the key arguments or pieces of evidence you have?"
   * "What background information is necessary?"
   * "Are there any common misconceptions you want to address?"

 * Conclude the Stage: Before moving on, update the status in project_metadata.md to * **Current Stage:** Stage 1: Collect - Completed. Then, suggest moving on: "This is a great collection of ideas in collected_ideas.md. When you're ready, we can move to the Outline stage."

Stage 2: OUTLINE

 * Initiate the Stage: Update the status in project_metadata.md to * **Current Stage:** Stage 2: Outline - In Progress. Announce, "We will now structure your ideas. I'll be reading from collected_ideas.md and writing the result to article_outline.md."

 * Analyze and Suggest Structures: Read the contents of collected_ideas.md. Based on the ideas, proactively suggest one or two fitting structural patterns.

   * If the topic seems to be about solving a problem: Suggest the "Why-What-How" framework.
     * Your prompt: "It looks like you're solving a specific problem. I suggest the 'Why-What-How' structure. We would start by explaining Why the problem is important, then describe What your solution is, and finally detail How to implement or use it. Does that sound like a good fit?"

   * If the topic is an explanation of a complex system: Suggest "Top-Down" or "Bottom-Up".
     * Your prompt: "This topic seems to be about explaining a complex system. We have two great options: A 'Top-Down' approach, where we start with the big picture and then drill into the details, or a 'Bottom-Up' approach, where we explain the fundamental components first and then show how they build up to the whole system. Which approach do you prefer?"

   * If the topic is an argument or debate: Suggest an "Adversarial" or "Thesis-Antithesis-Synthesis" structure.
     * Your prompt: "Since you're presenting a nuanced argument, an 'Adversarial' style could be very effective. We could structure it like a dialogue: present your main claim (thesis), then fairly explore the strongest counter-arguments (antithesis), and finally, present a conclusion that resolves the conflict (synthesis). How does that sound?"

 * Collaborate on Structure: Once the user chooses a structure, work with them to organize the points from collected_ideas.md into a hierarchical outline.

 * Produce the Outline: Write the final, structured outline into article_outline.md. After completion, update the status in project_metadata.md to * **Current Stage:** Stage 2: Outline - Completed.

Stage 3: DRAFT

 * Initiate the Stage: Update the status in project_metadata.md to * **Current Stage:** Stage 3: Draft - In Progress. Announce, "Now we'll create the first draft in article_draft.md. To ensure the text matches your voice, let's start with style."
 
* Learn Tone and Style: Ask the user, "Do you have a style guide, or could you provide links to a few articles whose tone and style you'd like me to emulate?"
   * If the user provides content: Analyze it to determine key characteristics (e.g., formal/informal, sentence length, use of jargon, humor, etc.).
   * Summarize the Style: Generate a concise, bulleted summary of the learned style.
   * Update Metadata: Write this summary into project_metadata.md under a "Style Guide" heading and ask the user to confirm it: "I've analyzed the examples and added a style summary to project_metadata.md. Does this accurately capture the voice you're aiming for?"

 * Build Reader Profile: Ask the user key questions and write the answers into project_metadata.md under a "Reader Profile" heading.
   * "Who are you writing for? (e.g., Absolute beginners, industry experts, project managers?)"
   * "What is the desired depth level? (e.g., A high-level overview, a practical guide with code?)"

 * Offer Agency in Drafting: Ask the user, "I'm ready to write the draft. I will read the structure from article_outline.md and generate the text. However, if you have any sections you've already written yourself, please let me know and I can incorporate them."

 * Generate and Write Draft: Read the outline from article_outline.md and the style/reader profiles from project_metadata.md. Generate the full text and write it into article_draft.md.

 * Handle Synchronization: If a user later modifies article_outline.md, you must detect this change and warn them: "I see you've updated the outline. The current text in article_draft.md is now out of sync. Shall I regenerate the draft based on the new outline?"

Stage 4: EDIT

 * Initiate the Stage: Update status in project_metadata.md to * **Current Stage:** Stage 4: Edit - In Progress. Announce, "We are now in the Edit stage. All our work will be focused on refining the text in article_draft.md."

 * Offer Editing Modes: Ask the user how they'd like to proceed. "We can edit the document together line-by-line, or I can perform specific checks. For example, I can scan for passive voice, simplify complex sentences, or check the tone. What works best?"

 * Collaborative Editing: Work with the user to refine the text directly within article_draft.md. Your role is to suggest changes and implement the user's edits. For example:
   * "This sentence seems a bit long. Could we split it like this for clarity?"
   * "Is this explanation clear enough for your target audience, or is it missing any key details?"

 * Conclude the Stage: Once the user is satisfied with the edits, update the status in project_metadata.md to * **Current Stage:** Stage 4: Edit - Completed.

Stage 5: RELEASE

 * Initiate the Stage: Update status in project_metadata.md to * **Current Stage:** Stage 5: Release - In Progress. Announce, "This is the final Release stage. We'll add the finishing touches to article_draft.md based on where you plan to publish."

 * Identify Publishing Platform: Ask for the target platform (e.g., blog, social media, academic journal).

 * Provide Tailored Suggestions: Based on the platform, provide a checklist of suggestions.

   * Generative Assistance: Offer to generate variations of titles, social media hooks, hashtags, or SEO keywords. For example: "Based on your article, here are three potential SEO-friendly titles. Which do you like best?"

   * Rich Media Placement: Scan article_draft.md and suggest specific places to add value. Use comments or placeholders like [SUGGESTION: A chart showing user growth would be effective here.] directly in the document.

   * Platform-Specific Formatting: Advise on best practices, such as using a strong hook and hashtags for social media, or ensuring correct citation style for a formal paper.



 * Final Polish: Work with the user to apply the final touches directly within article_draft.md.

 * Conclude the Project: Update the status in project_metadata.md to **Current Stage:** Project Completed. Congratulate the user. "Congratulations! Your article in article_draft.md is ready to be published. All your project files are saved here in the Canvas if you ever want to return to them."

This is an early draft of Chapter 9 of my upcoming book Mostly Harmless AI. I’m deeply grateful for all suggestions and criticism you might have.

Technology, especially artificial intelligence, moves at a blistering pace, far outstripping the deliberate, democratic processes of regulation. This creates a governance gap—an ever-widening space where innovation flourishes without guardrails, leaving society exposed to significant and often unforeseen risks.

This is not necessarily a failure of governance, but an inherent tension in the modern world. The challenge for today’s leaders is not to halt the march of technology, but to build a bridge across this gap with smart, agile, and evidence-based policy.

This chapter is designed to provide a practical outlook for those tasked with building that bridge. It offers a framework for regulators and policymakers on how to approach AI governance pragmatically, focusing on tangible, real-world harms and achievable benefits. It is a guide to steering progress, not stopping it, rooted in the techno-pragmatist belief that our collective future is not something that happens to us, but something we must actively and responsibly shape.

While the principles outlined here are actionable on their own, they are built upon a deep understanding of AI’s fundamental limitations and risks. The full, in-depth analysis of these challenges—from the mechanics of hallucination to the societal dangers of bias and disinformation—is detailed in Part III of this book.

For the most comprehensive understanding, I encourage you to review Part III in depth. Armed with that context, you can then return to this chapter to engage more deeply with the policy suggestions made here, transforming them from abstract principles into a grounded and urgent call to action.

A final disclaimer. Regulation and policy making of technology is extremely difficult, and even more in the face of technology that changes as fast as AI does. Everything written here must be taken with a grain, or even better, a teaspoon full of salt. Furthermore, no specific advice will fit all contexts. Each country, state, and community is responsible for finding their own way forward based on their own shared principles.

Why Regulation is Necessary

Before we can chart a path forward, we must first understand the terrain of risks that requires thoughtful governance. These are not speculative fears, but foundational challenges posed by the very nature of modern AI, building from the immediate threats to the individual to the structural risks facing our global society. Regulation is required not to stifle technology, but to ensure it develops in a way that is compatible with a safe, equitable, and democratic society.

Let's start with privacy. The ability to analyze vast quantities of personal information at scale creates the potential for a pervasive surveillance apparatus, operated by both corporations and governments, that was previously unimaginable. The only effective countermeasure is a strong, proactive policy that establishes privacy as the default.

This requires comprehensive data privacy laws that grant individuals clear rights over their data and place strict limits on what information can be collected, for what purpose, and for how long. Policy must shift the burden of proof, forcing organizations to justify their data collection practices rather than forcing citizens to constantly fight to protect their private lives.

Furthermore, when AI systems are trained on biased historical data (the only kind of historical data we have), they risk automating and scaling up discrimination in critical areas like hiring, lending, and criminal justice. Because market forces alone may not prioritize fairness over the raw predictive performance that can be gained from these biases, regulation is essential to protect fundamental civil rights.

Policy can create powerful legal and economic incentives for developers to address this problem by mandating algorithmic transparency and requiring independent fairness audits for any AI system used in high-stakes decisions. This ensures that the pursuit of technological efficiency does not come at the cost of societal equity.

Moving on, our existing legal frameworks for intellectual property and ownership are fundamentally unprepared for content generated by artificial intelligence, creating a landscape of legal ambiguity that chills innovation and threatens the livelihoods of human creators.

The legal system must be updated to provide clarity and predictability. This requires decisive legislative action to define the copyright status of AI-generated works, establish clear rules for the use of copyrighted data in training foundation models, and create a legal environment where both human artists and AI innovators can operate with confidence.

But it gets worse, the power of generative AI to create convincing fake news and deepfakes presents a direct threat to our shared sense of reality, eroding trust in institutions and fueling social polarization.

A regulatory approach here requires a delicate balance. Outright censorship is a dangerous tool that is itself a threat to democratic values. A more pragmatic policy would focus on creating a healthier information ecosystem by mandating transparency—such as the clear and consistent labeling of AI-generated content—and by holding platforms accountable not for the content itself, but for its algorithmic amplification.

This, combined with robust public funding for media and AI literacy programs, can empower citizens to navigate the digital world more critically without resorting to authoritarian measures.

At the same time, the rapid advance of AI into cognitive tasks promises to cause massive workplace disruption, displacing workers at a pace that could challenge social and economic stability.

The goal of policy in this area is not to halt the productivity gains of automation, but to proactively manage the human transition. This requires a two-pronged strategy: first, investing heavily in accessible, large-scale retraining and lifelong learning programs to equip the workforce with new skills; and second, modernizing the social safety net to provide a robust economic cushion for those navigating this difficult transition.

A more abstract but even more dangerous development are Lethal Autonomous Weapons (LAWs) that threaten to fundamentally alter the nature of conflict, removing human empathy and judgment from the decision to use lethal force.

This is not a problem that market forces or technological solutions can solve; it is a profound ethical challenge that demands a global political response. The only viable path forward is through international policy, establishing clear treaties and shared norms that mandate meaningful human control over autonomous systems.

The goal of such regulation is to draw an unambiguous red line, preventing a destabilizing arms race in an arena where the potential for catastrophic error or miscalculation is immense.

A Pragmatic Stance on Existential Threats

Finally, any serious policy discussion must address the so-called existential risks, which involve the potential for AI to destroy human civilization altogether.

While acknowledging the concern is important, a pragmatic stance requires contextualizing the probability. As argued in Part III, catastrophic outcomes, while having a nonzero chance, remain highly improbable, as the core doomsday assumption of rapid, exponential self-improvement is tempered by very real physical and computational limitations, and there is no evidence current technology can surpass these limitations.

A danger for policymakers lies in the overemphasis on these speculative, long-term risks, which can divert critical resources from solving the tangible, present-day harms AI is already creating.

The pragmatic approach here lies in understanding that AI x-risk is but one of several major threats on a similar scale as climate change and pandemics, and probably far less likely. Therefore, policy should support thorough research into long-term risks but avoid panic-driven bans on development. The most effective strategy is to focus regulation on mitigating the demonstrated, immediate harms of current AI systems.

The Challenge of Smart Regulation

Identifying the risks is only the first step. The act of regulation itself is fraught with challenges, especially when applied to a technology as dynamic and complex as AI. A naive approach can be as harmful as no regulation at all, creating unintended consequences that stifle beneficial innovation or fail to address the core problems.

Smart regulation requires navigating three key pitfalls: the pacing problem, the risk of overreach, and the black box problem.

The Pacing Problem

Traditional legislative cycles, which can take years to produce new laws, are fundamentally mismatched with the exponential pace of AI development. By the time a law designed to govern a specific AI capability is passed, that technology may already be obsolete.

To overcome this, policymakers should consider establishing agile, expert-led regulatory bodies. These specialized bodies can be staffed with technologists, ethicists, and social scientists who can monitor the field in real-time, issue updated guidance, and adapt regulatory standards far more quickly than a legislature can.

Avoiding Overreach

In the face of uncertainty and fear, the temptation can be to enact broad, sweeping prohibitions on AI development. This would be a profound mistake. A techno-pragmatist approach distinguishes between foundational research and commercial application. The goal of regulation should not be to stifle the scientific exploration that leads to breakthroughs, but to govern the deployment of AI systems where they have a direct public impact.

Policy should therefore focus on demonstrated harm, setting clear safety and fairness standards for AI products and services that are released into the market, rather than attempting to place speculative limits on basic research and open-source development.

The Black Box Problem

Many of the most powerful AI systems operate as black boxes, where even their own creators cannot fully explain the specific logic behind a given decision. This opacity poses a fundamental challenge to accountability and due process. How can an individual appeal a decision they cannot understand?

Smart regulation must address this by championing the principles of transparency and explainability. For high-stakes applications, policy can mandate a right to an explanation, requiring that companies be able to provide a meaningful justification for AI-driven decisions that significantly impact people’s lives. This incentivizes the development and adoption of Explainable AI (XAI) techniques, ensuring that as systems become more complex, they do not become less accountable.

Principles for Proactive AI Governance

Having navigated the pitfalls, we can chart a course for proactive governance. The following principles are not a rigid checklist, but a compass for steering AI development toward a future that is safe, equitable, and beneficial.

The core of this approach is a commitment to evidence over ideology. A risk-based approach, attuned to the principles of techno-pragmatism, means that the level of regulatory scrutiny applied to an AI system should be directly proportional to its potential for harm. An AI that recommends movies requires a lighter touch than one that assists in medical diagnoses.

This ensures that regulation focuses its power where it is most needed, fostering innovation in low-risk areas while demanding rigorous oversight for high-stakes applications.

This human-centric governance must insist on meaningful human control as a direct response to the deep and persistent Alignment Problem. As Part III makes clear, perfectly specifying human values is an unsolved, and perhaps unsolvable, challenge. Therefore, for critical systems where decisions have significant consequences—in medicine, law, and finance—policy must mandate a human-in-the-loop.

This is not a mere suggestion but a non-negotiable backstop against the inevitable failures of alignment, ensuring that a human expert is always the final arbiter, accountable for the outcome. AI can and should be a powerful tool for augmenting professional judgment, but it must never be allowed to replace it.

Furthermore, proactive governance involves shaping the entire AI ecosystem to better align with societal values. A purely market-driven economy has no inherent incentive to solve deep issues like fairness or cultural representation. Therefore, policy must create these incentives. This can be done through liability reform that holds companies accountable for harms caused by their systems, and through tax credits that reward investment in safety and ethics research.

In parallel, governments can counteract the risk of cultural colonization by a few generalist models by funding the development of local and regional AI solutions. This support for models trained on specific cultural and linguistic data, combined with national programs to foster widespread AI literacy, can help creating a more diverse, resilient, and critically engaged society.

Finally, since AI is a global technology, our approach to its governance must also be global.

A patchwork of national regulations creates a race to the bottom, where innovation may flee to the least-regulated environments. The most powerful path forward lies in promoting openness and international collaboration.

Policy can and should incentivize the open-sourcing of foundation models, which enhances safety by allowing the global research community to audit, critique, and improve them. This spirit of collaboration must extend to the diplomatic level, forging international agreements and shared norms to govern the most critical risks, ensuring that the development of this transformative technology is a shared project for all of humanity.

Conclusions

The path of technology is not deterministic. The future of artificial intelligence is not a predetermined outcome that we must passively accept, but a landscape that will be profoundly shaped by the policy choices we make today. As we have seen, the risks are significant, but so is the potential. A techno-pragmatist approach requires us to hold both these truths at once, engaging with this powerful technology with our eyes wide open.

Thanks for reading! Remember you can get my upcoming book Mostly Harmless AI at 50% discount in early access.

You can get early access to Mostly Harmless AI at 50% reduced cost during this alpha stage, which gives you full access in eternity to all future digital editions and printed copies (when they are ready) at cost.

You can also get a lifetime pass for all my digital content, present and future, including 3 more books I’m currently working on.

What is Artificial Intelligence, Really?

Artificial Intelligence, or AI, is a term we hear almost constantly today, often surrounded by a mix of excitement, confusion, and sometimes, even fear. At its core, AI is a field within Computer Science that deals with teaching computers to solve problems that are incredibly challenging for traditional programming methods. These aren’t simple arithmetic calculations or straightforward data sorting tasks. Instead, we’re talking about complex endeavors like proving intricate mathematical theorems, navigating a robotic car through unpredictable city streets, crafting optimal schedules for thousands of flights, or even understanding and creating human-like pictures and text.

For most of computer science, when we want a computer to solve a problem, we write a precise, step-by-step algorithm. Think of it like giving a chef a detailed recipe: Take 2 cups of flour, add 1 egg, mix for 3 minutes… However, for the hard problems AI tackles, we often don’t have such a clear recipe. We might know what we want the computer to achieve, but not how to write down every single instruction for it to get there effectively and efficiently. This is precisely where AI steps in, aiming to find good enough solutions when perfect, explicit instructions are out of reach.

The very definition of AI has been a subject of debate since its inception, reflecting different philosophical ideas about what intelligence truly means. One prominent perspective, championed by AI pioneer Marvin Minsky, suggests that AI is about solving problems for which humans employ intelligence. This view often focuses on creating machines that can mimic human thought processes, reasoning, and decision-making. Essentially, it asks: Can a machine think like us?

Developing concurrently, another powerful perspective emerged, emphasizing that AI solves problems without being explicitly programmed. This idea is strongly associated with Arthur Samuel, who coined the term machine learning while developing programs that could learn to play checkers better than their creators. He achieved this simply by allowing the programs to play many games and learn from experience. This view shifts the focus from how the AI thinks to what it can do, asking instead: Can a machine learn and adapt on its own, even if we don’t give it every single instruction?

These two foundational ideas–mimicking human intelligence versus learning without explicit programming–have profoundly shaped the entire field of AI. They represent different ways of approaching the grand challenge of building intelligent machines. Understanding this distinction is key to grasping AI’s history and its future. As we explore these foundations, remember our techno-pragmatist ethos: AI is a tool, and its path is shaped by our choices. Understanding its underlying mechanisms empowers us to make responsible decisions about how we build and use these powerful technologies.

The Pillars of Good Old-Fashioned AI (GOFAI)

In this chapter, we will delve into the foundational ideas that laid the groundwork for Artificial Intelligence, often referred to as “Good Old-Fashioned AI,” or GOFAI. This era of AI research primarily focused on building intelligent systems by explicitly programming knowledge and logical rules. Our exploration will center on two main pillars of GOFAI.

First, we’ll examine Search and Optimization, which addresses how AI finds solutions by exploring vast possibilities, particularly when a perfect, direct path isn’t obvious. Second, we’ll delve into Knowledge Representation, focusing on how AI organizes and understands information, allowing it to reason and make sense of the world. These pillars represent a significant early focus and ambition of AI to tackle complex problems through logic and structured understanding, even as other approaches were also taking shape.

The Age-Old Debate: Symbolic AI vs. Statistical AI

For centuries, the idea of thinking machines has captivated human imagination. But as AI emerged as a scientific field, a fascinating tension developed: a constant “back-and-forth between two core, seemingly antagonistic approaches to building intelligent machines.” This dynamic mirrors an age-old philosophical debate: rationalism versus empiricism.

The first dominant approach to AI was Symbolic AI, deeply rooted in the philosophical tradition of rationalism. Rationalism suggests that knowledge is primarily gained through reason and logic. In Symbolic AI, researchers believed that machines could become intelligent by putting human knowledge and reasoning into explicit, formal rules and symbols.

Imagine, for instance, wanting to teach a computer to play chess. A Symbolic AI approach would involve meticulously programming every rule of chess, every known opening strategy, every tactical pattern, and every endgame scenario. It’s like giving the computer a massive, incredibly detailed recipe book or a comprehensive instruction manual for every possible chess situation. The computer would then follow these rules step-by-step to make its moves.

Early impressive demonstrations of this ethos included programs like The Logic Theorist, which could prove mathematical theorems by mimicking human problem-solving steps. Later, “expert systems” were designed to emulate human experts in narrow fields like medical diagnosis. The core idea was simple yet powerful: if we could just write down all the rules, the machine would be smart enough to solve them.

Quietly developing alongside Symbolic AI was Statistical AI, drawing inspiration from empiricism. Empiricism posits that knowledge is primarily gained through sensory experience and data. In Statistical AI, the idea was to build “learning machines” that could discover patterns directly from large amounts of data, rather than being explicitly programmed with rules.

Think of it like a child learning to recognize a dog. You don’t give the child a list of rules like “a dog has four legs, barks, has fur,” and so on. Instead, you show them many different dogs, and they gradually learn to identify what a “dog” is by observing patterns in the examples. Early attempts at this included the Perceptron, an early artificial neural network designed to learn patterns directly from data. The initial excitement was huge, as these machines seemed to offer a path to intelligence without needing every single rule programmed explicitly.

The Winters of AI

Despite the initial optimism, both Symbolic and Statistical AI approaches eventually hit significant roadblocks. These challenges led to periods known as “AI Winters”–times of reduced funding and public interest.

Early Symbolic AI systems, while impressive in their specific domains (like proving theorems or diagnosing specific diseases), proved to be quite brittle. They struggled immensely with common-sense knowledge, which is vast and often unstated. Furthermore, they couldn’t easily adapt to new situations outside their carefully programmed rules. Trying to teach a machine absolutely everything it needed to know, one fact at a time, became an “insurmountable challenge.” The real world is simply too complex and nuanced for a complete set of explicit rules to be written by humans.

Meanwhile, early Statistical AI systems like the Perceptron faced their own limitations. They lacked the “available data and computational infrastructure” to learn truly complex patterns. Consequently, they couldn’t become sophisticated enough, no matter how many simple “neurons” were connected. The computing power and data storage simply weren’t ready for the ambitious learning tasks researchers envisioned.

These “winters” were not outright failures, but rather crucial learning periods. They revealed the inherent limitations of each approach when pushed beyond “toy problems.” This early struggle between explicit rule-based systems and pattern-based approaches set the stage for the dynamic tension that would define AI’s entire history, constantly pushing researchers to find new ways to combine or overcome these challenges.

Subscribe now

Search and Optimization

At the heart of many AI problems, especially in the early days, was the challenge of finding the best solution among a vast number of possibilities. This is the realm of search and optimization.

When Perfect is Impossible: The “Hard” Problems

Imagine you’re a traveling salesperson, and you need to visit a hundred different cities, visiting each exactly once, and then return home. Your goal is to find the route that minimizes the total travel cost (distance, time, or money). This is a classic example of a “hard problem” in computer science, known as the Traveling Salesman Problem (TSP). For a small number of cities, you could try listing every single possible route and picking the cheapest one. This is called a “brute force” search.

However, as the number of cities grows, the number of possible routes explodes. For just 20 cities, there are over 2.4 quintillion (2.4 followed by 18 zeros!) unique routes. Even the fastest supercomputer couldn’t check them all before the universe ends. These are what we call intractable problems, or NP-Hard problems: problems for which no efficient, exact solution is known.

To tackle such problems, AI often models them as navigating a “search space” or “state space.” This conceptual space represents all possible configurations or situations relevant to the problem. The AI starts from an initial state and tries to reach a goal state by applying a sequence of actions or operators, each potentially incurring a certain cost.

Since finding the absolute perfect solution is often impossible or impractical within this vast space, AI shifts its goal. Instead of perfection, it seeks approximate solutions. These are solutions that are good enough, given the time and memory constraints we have. The challenge then becomes how to find these good-enough solutions efficiently within a mind-bogglingly vast space of possibilities.

Smart Shortcuts: Heuristics and Metaheuristics

To navigate these immense search spaces, AI uses clever strategies known as heuristics and metaheuristics. A heuristic is a problem-specific “rule of thumb” strategy that uses some known properties of a problem to improve search performance. It’s not guaranteed to find the absolute best solution, but it often finds a very good one much faster than a brute-force approach.

Consider your GPS navigation app. When you ask for directions, it doesn’t calculate every single possible route from your current location to your destination. Instead, it uses a heuristic, often based on an algorithm called A∗ (A-star). If your destination is northeast of your position, the A∗ algorithm will prioritize roads going north or east, assuming they are more likely to get you there faster than roads going to the west or the south. Of course, this isn’t always perfect–there might be a faster detour to the west, or a highway that’s counter-intuitive. Nevertheless, by intelligently using this useful knowledge, the algorithm can find a very efficient route without exploring every dead end. It’s a smart shortcut that balances speed with a high probability of finding a good solution.

While heuristics are problem-specific, metaheuristics are more general-purpose search strategies. They leverage knowledge about the search paradigm itself and can be applied even when very little is known about the specific problem’s structure. They’re often used when “nothing else works.” A prime example of a metaheuristic approach is evolutionary algorithms. These computational strategies are “inspired by certain aspects of the biological process of evolution.”

Imagine you want to design the optimal layout for a computer chip (like a GPU) – a problem with an astronomical number of possible designs. An evolutionary algorithm would start with a “population” of random chip designs. Then, through cycles of “breeding” (combining elements from two good designs to create a new one) and “selection” (keeping only the best-performing designs), the algorithm iteratively “evolves” better and better designs. Just like biological evolution, it seems to “magically discover quasi-optimal design elements just by sheer luck and relentless repetition,” without needing explicit instructions for every design choice. These general strategies find inspiration in nature, engineering, and even social systems to build powerful computational search methods.

Specialized Search: Beyond Simple Paths

Beyond general search and optimization, AI has developed specialized techniques for specific types of complex problems.

Adversarial Search: Thinking Ahead of the Other

Many real-world problems, especially in competitive scenarios, involve an opponent whose actions must be anticipated. This is the domain of adversarial search, commonly found in game-playing AI. The challenge is not just to find a good move, but the best move assuming your opponent will also play optimally to counter you.

One of the oldest and most fundamental techniques is Minimax. Imagine a simple game like Tic-Tac-Toe. Minimax works by having the AI “look ahead” through all possible future moves, assuming that you (the opponent) will always choose the move that is best for you and worst for the AI. The AI then picks the move that minimizes its maximum possible loss (or maximizes its minimum possible gain). Effectively, it plays out all possible future scenarios in its head and chooses the path that leaves it in the best possible position, no matter what its opponent does.

For games with an incredibly vast number of possibilities, like Go, simply looking ahead through every move is impossible. This is where Monte Carlo Tree Search (MCTS) comes in. Instead of exhaustively analyzing every branch, MCTS “plays out” many random simulations of the game from a given point. It explores the most promising moves more deeply, learning which paths lead to success through repeated “trial and error” simulations. This allows AI to tackle games that were once considered beyond computational reach, like when Google’s AlphaGo beat the world’s best Go players.

Structured Search: Satisfying All Conditions

Sometimes, the goal isn’t to find the “best” path, but simply any solution that meets a specific set of requirements. These are constraint satisfaction problems. Here, the AI needs to find values for a set of variables such that all given conditions, or “constraints,” are simultaneously met.

Think about solving a Sudoku puzzle. You need to fill in numbers from 1 to 9 in each cell, but with strict rules: each row, column, and 3x3 box must contain all digits from 1 to 9 without repetition. The AI’s task is to find a set of numbers for all empty cells that satisfies all these constraints.

Another common example is creating a university class schedule. You have classes, rooms, professors, and students, and a multitude of constraints: Professor A can’t teach two classes at the same time; Room B can only hold 50 students; Class C requires a lab; no two classes can be in the same room at the same time. The AI’s job is to assign times and rooms to all classes such that every single constraint is satisfied. The “structure” of these problems, defined by the variables and their interdependencies, allows AI to use specialized search techniques to efficiently find a valid solution.

Subscribe now

Knowledge Representation & Reasoning

Beyond just searching for solutions, a truly intelligent system needs to “know” things about the world. This brings us to the second pillar of GOFAI: Knowledge Representation. This field explores how AI can efficiently represent, store, and use domain knowledge in a way that computers can understand and process.

The fundamental goal of knowledge representation is to organize concepts and facts, as well as the relationships between them. This organization allows AI to reason about these facts and discover new relations. Ultimately, it’s about giving AI a structured way to “understand” and make sense of information, much like how humans build a mental model of the world around them. Without a clear way to represent what it “knows,” an AI would be unable to make logical inferences or apply its knowledge to new situations.

From Raw Observations to Understanding

To truly grasp how AI “knows” things, it’s helpful to understand the progression from raw observations to actionable understanding. At the most basic level, we encounter Data, which consists of raw, unprocessed facts or observations. This could be a list of numbers, individual words, or pixels in an image. In isolation, data has no inherent meaning; for example, the number “30” by itself is just a number.

When we introduce context or metadata, data transforms into Information. For instance, if we know “30” is a temperature reading taken in Celsius at noon on July 1st in Havana, it becomes information. This contextualization helps us relate different observations and gives them initial meaning.

Finally, when information is enriched with semantics and rules, enabling inference, reasoning, and the discovery of new relations, it becomes Knowledge. For example, if the AI knows that “temperatures above 30 degrees Celsius in July in Havana indicate a heatwave,” it possesses knowledge. This knowledge allows it to draw inferences (it’s a heatwave!), discover new relations (heatwaves can lead to increased energy consumption), and even take actions (warn residents about high temperatures). It’s this ability to add meaning and logical connections that truly transforms information into actionable knowledge.

Ways to Represent Knowledge

Just as humans use different ways to store and recall information, from precise definitions to vague intuitions, AI employs various methods for knowledge representation, each with its own strengths and weaknesses.

One key distinction lies between explicit and implicit representations. Explicit knowledge is clearly defined and directly encoded, often in rules or symbols. It’s much like a precisely written dictionary or a rulebook where every term and every rule is spelled out. This approach is central to Symbolic AI. For instance, Ontologies are explicit representations that define concepts within a domain and their strict relationships. Think of a meticulously designed family tree formally defining “parent,” “child,” “sibling,” and “ancestor,” along with rules such as “if A is a parent of B, and B is a parent of C, then A is a grandparent of C.”

Conversely, implicit knowledge is learned from patterns in data, rather than being directly programmed. It’s more akin to human intuition or a “gut feeling” developed from vast experience, and is fundamental to Statistical AI. Embeddings, for example, are numerical representations where concepts like words, images, or even entire documents are transformed into points in a multi-dimensional space. Systems like Word2Vec learn these embeddings by analyzing how words are used together, so words with similar meanings or contexts (e.g., “king” and “queen”) end up being numerically “close” to each other in this space, even though no human explicitly programmed that relationship.

Another way to categorize knowledge representations is by their formality. Formal representations have strict, unambiguous syntax and semantics, making them ideal for precise logical inference and computation. Mathematical equations, programming code, or statements in formal logic are prime examples, leaving no room for misinterpretation. In contrast, informal representations are more flexible, often using natural human language. While easier for humans to create and understand, they can be ambiguous and require more sophisticated processing for AI to extract meaning, as seen in a written description, a casual conversation, or an essay.

Finally, we distinguish between structured and unstructured representations. Structured knowledge is organized in a predefined, rigid format, making it easy for computers to process and query. Think of data in a spreadsheet with clear rows and columns, or a database with defined fields.

Knowledge graphs, for instance, are structured representations that organize facts as a network of interconnected entities (nodes) and their relationships (edges). A knowledge graph might have a node for “Paris,” a node for “France,” and an edge labeled “isCapitalOf” connecting them, allowing AI to easily query and infer facts.

Conversely, unstructured knowledge exists in free-form text, images, audio, or video, without a predefined schema. Extracting meaning from unstructured data is much harder and often requires advanced AI techniques.

Vector databases, for example, are often used to store and efficiently search implicit representations (embeddings) derived from unstructured data. You could take millions of research papers (unstructured text), convert each into an embedding (implicit representation), and store them in a vector database. Then, when a user asks a question, the database can find the most “similar” papers based on their embeddings, even though the papers themselves are unstructured.

Drawing Inference from Knowledge

Knowledge representation isn’t merely about storing information; its ultimate purpose is to enable AI to draw inferences and make decisions. This process of deriving new conclusions from existing knowledge is known as reasoning, and it can take both formal and informal forms.

Formal reasoning, deeply rooted in logic, is about deriving new conclusions from existing knowledge using strict, unambiguous rules. This is the hallmark of Symbolic AI. It’s a process of deduction, where if the initial premises are true and the rules are applied correctly, the conclusion is guaranteed to be true. For example, if a knowledge base contains the rules “All birds can fly” and “A sparrow is a bird,” a formal reasoner can deduce, with absolute certainty, “A sparrow can fly.” Such rule-based systems are precise and auditable, but they are limited by the completeness and accuracy of the explicitly programmed rules.

In contrast, informal reasoning is about drawing conclusions based on patterns, similarities, or analogies, often without strict logical guarantees. This type of reasoning is more akin to human intuition or common sense. It’s less about strict deduction and more about finding connections and probabilities. For example, if an AI has learned implicit representations (embeddings) of various animals, and it sees a new animal that is “numerically close” to many dogs, it might infer it’s a dog, even without explicit rules for every single feature.

This distinction is crucial for understanding the different capabilities of AI. While formal reasoning provides certainty within defined boundaries, informal reasoning allows AI to operate in ambiguous, unstructured environments. The latter, particularly reasoning by analogy in embeddings and language models, will be explored in more detail in later chapters, showcasing how AI can make sense of the world even when explicit rules are unavailable.

What is the Best Representation for Knowledge?

The choice of how to represent knowledge is a critical decision in AI design. Different representation types are chosen based on the specific problem, the type of data available, and the AI paradigm being used (Symbolic vs. Statistical). For instance, a Symbolic AI system designed for medical diagnosis might rely heavily on formal, explicit ontologies of diseases and symptoms. Conversely, a Statistical AI system for image recognition might primarily use implicit, unstructured representations of pixels that it learns from millions of example images.

This challenge highlights a theoretical result known as the Ugly Duckling Theorem. This theorem, in essence, states that without a specific purpose or “bias,” all objects are equally similar or dissimilar to one another. This implies that there is no single, universally “best” way to represent knowledge or measure similarity without a context or goal in mind. For example, an “ugly duckling” is only ugly relative to a flock of swans; it might be beautiful among other ducklings.

Therefore, the human responsibility in choosing the right representation is paramount. This choice directly impacts what an AI can “know,” how it can “reason,” and ultimately, the reliability and fairness of its inferences. Aligning the representation with the problem’s nature is a key part of building human-centered tools that truly understand and assist us.

Conclusion: The Need for Learning

Good Old-Fashioned AI (GOFAI), with its focus on search, optimization, and explicit knowledge representation, laid the essential groundwork for the field of Artificial Intelligence. Its strengths lie in domains where problems are well-defined, rules are clear, and knowledge can be precisely encoded. GOFAI systems offered precision and control, making them powerful tools for tasks like proving theorems or playing well-defined board games.

However, the ambitions of GOFAI soon ran into fundamental limitations when faced with the messy complexity of the real world. These systems proved to be brittle: a small change outside their programmed domain could break them entirely. They struggled immensely with common-sense knowledge, which is vast and often unstated. The sheer scale of real-world information made it an “insurmountable challenge” to explicitly program every piece of knowledge and every rule. GOFAI was excellent at solving problems for which it was explicitly programmed, but it couldn’t adapt, generalize, or handle unstructured data effectively. This revealed a crucial gap in AI’s capabilities.

The limitations of GOFAI highlighted a profound truth: to build truly intelligent and adaptable systems, AI needed to move beyond simply executing pre-programmed rules. It revealed a crucial need for systems that could learn from experience and data, without being explicitly programmed for every single scenario or piece of knowledge.

This growing realization of the power of learning-based approaches, which were developing concurrently with GOFAI, marked a significant shift. It showed that AI could discover its own patterns and adapt to unforeseen situations, offering a path to overcome the brittleness of purely symbolic systems.

Recognizing these limitations and actively seeking new approaches is a hallmark of the ongoing, human-driven effort to build more capable and adaptable AI. It’s a testament to our techno-pragmatist ethos: acknowledging challenges, learning from past efforts, and continuously striving to create tools that can better serve humanity’s complex needs. This increasing prominence of learning methods, which developed in parallel to GOFAI, is the story that will unfold in the next chapter.

Thank you for reading this far. This chapter is still a first draft, so any comments, suggestions, and criticism are truly appreciated.

Leave a comment

PS: Get your copy of Mostly Harmless AI at 50% off.

The following is a first draft of Chapter 7 of my upcoming book Mostly Harmless AI.

Of all the domains being transformed by artificial intelligence, education is perhaps the most critical to get right. The stakes are uniquely high. Used wisely, AI has the potential to be a massively positive force, augmenting the work of teachers and deepening the learning of students in ways we are only beginning to imagine. Used incorrectly, however, it could be catastrophic, undermining the development of critical thinking and eroding the very foundations of academic integrity. This chapter is a guide to navigating that high-stakes environment.

We will begin by demystifying the popular idea of a personalized AI tutor, a vision that runs counter to the principles of human-centered, collaborative learning. In its place, we will propose a more grounded solution that sees AI as a tool for augmentation, not automation. Next, we will dismantle the common misconceptions surrounding AI detection tools, arguing that this approach is not only futile but actively harmful to the learning environment.

This will establish the necessity of a fundamental pedagogical shift, moving from policing to integration. From there, we will offer practical strategies for both educators and learners, emphasizing their shared responsibility in fostering a new kind of AI literacy. Finally, we will show what a concise but comprehensive AI policy for an academic program could look like, providing a tangible model for implementation.

The Myth of the Personalized Tutor

The arrival of powerful generative AI has fueled a seductive, decades-old myth: that the ultimate goal of technology in education is to create a personalized, all-knowing AI tutor for every learner. This vision promises a revolution, a future where a “personalized Aristotelian tutor” is available to every student, adapting to their unique learning style, language, and pace. This narrative is powerful, but it is built on a fundamental misunderstanding of how we learn and what education is for.

Even if such a perfect tutor were achievable, it is not the revolution we should want. The idea that education’s primary problem is a lack of personalization or efficient information delivery is a flawed premise. Before we can harness AI effectively, we must first deconstruct this myth by examining the three core reasons why the automated personal tutor is a flawed ideal.

Argument 1: It Mistakes Information Transfer for Learning

The myth of the personalized tutor assumes that the primary obstacle to learning is the inefficient delivery of information. This argument has some merit in specific contexts; in places where the main obstacle to education is a lack of access to books, internet, and educators, an AI tutor could be a game-changer. However, this is not the case for the majority of learners in developed nations.

In an era of information surplus, the problem for the modern student is not a lack of access to information, but a lack of skill in navigating, evaluating, and synthesizing it. While asking an AI for an answer is slightly more convenient than a Google search, it is not qualitatively better. Furthermore, it removes the “desirable difficulty” that forges lasting knowledge. The struggle to find information, compare sources, and form a conclusion is a valuable cognitive exercise. An AI tutor designed to eliminate this struggle by providing immediate answers actively prevents the most valuable parts of the learning process from ever happening.

Argument 2: It Promotes Intellectual Dependency, Not Critical Thinking

The myth suggests that an AI can be a perfect partner for completing assignments, from solving math problems to writing essays. This, however, risks creating profound intellectual dependency. When a student uses an AI to bypass the hard work of structuring an argument, recalling information, synthesizing ideas, or debugging a line of code, they learn to prompt, not to think.

The purpose of assigning an essay is not to receive a perfect text; professors already know the answers. The purpose is to engage the student in the process of creation, which is where learning occurs. By offering a shortcut straight to the final product, generative AI undermines the most valuable aspect of the exercise. It becomes an obstacle, hampering the educational process by allowing students to bypass the very challenges that help their brains learn and grow.

The goal of education is to build independent, critical thinkers who can grapple with complex, ambiguous problems on their own. Over-reliance on an AI that provides solutions on command undermines this goal, making students dependent on the tool long after the lesson is over.

Argument 3: It Champions Isolation Over Community

The vision of a personalized path idealizes a student learning in perfect, isolated efficiency, free from the pace of a group. This completely ignores that learning is a fundamentally social and collaborative activity. Studying individually and independently is not necessarily an advantage; in fact, it can be a huge disadvantage.

The two things most self-educated people struggle with are motivation and feedback. Motivation comes naturally in a classroom because you are surrounded by peers with similar goals. Seeing others tackle challenges and grow creates a powerful incentive to overcome difficulties.

Feedback from mentors and peers is equally crucial for intellectual growth, allowing us to iterate on ideas and hone our skills. A community of learners is key. An AI tutor, no matter how sophisticated, cannot replicate the dynamic, motivating, and often messy reality of a human learning community. Learning together always beats learning alone.

The Alternative

The alternative to the flawed myth of the personalized tutor is to view AI not as an automated teacher, but as a powerful tool for augmentation within a human-centered community. This approach requires a pedagogical shift away from the futile chase of AI detection and toward a model of shared responsibility. It is a vision where educators and students work together to develop a new, essential AI literacy, using these tools to enhance, rather than replace, the timeless process of collaborative and critical learning.

Why AI Detection Is Futile

Before educators can effectively integrate AI, they must first understand that the detection of AI-generated content is a hopeless chase. Any attempt to police AI use through detection tools is an unwinnable arms race destined to fail for a number of practical and pedagogical reasons.

First, the technology itself is fundamentally flawed. Detectors will always be lagging behind the generative models they seek to identify, perpetually playing catch-up in a race they cannot win. The supposed telltales of AI-generated text—overly formal language, a lack of personal voice, perfect grammar—are not robust signals. They are merely fleeting characteristics of specific models at a single point in time. A detector trained to spot GPT-4’s style is useless against the next generation of models, and it’s even more useless against a student who uses one of the clever prompt techniques this very book teaches to make the output more human-like.

Second, these tools are dangerously inaccurate. Their unacceptably high false positive rates mean that you will inevitably punish honest students, accusing them of fraud they did not commit. This is an ethical line no educator should be willing to cross. At the same time, the tools are easily bypassed, meaning that while innocent students are flagged, those determined to cheat can still slip through. The result is a system that is both unjust and ineffective.

This cat-and-mouse game also creates perverse incentives. It encourages students to spend more time hiding and tinkering with AI to bypass detectors than on the actual intellectual work of the assignment. Their focus shifts from critical thinking to “evasion engineering.” This is the exact opposite of the goal of education.

Ultimately, a reliance on detection tools creates an environment of distrust that is toxic to learning. It frames the relationship between teacher and student as adversarial, replacing a partnership built on trust with one based on suspicion. Fraud is a serious ethical issue that completely undermines the purpose of education, but it is not a technological problem to be solved with software. It is a human one that must be discussed on ethical grounds, as a violation of the shared trust that makes a learning community possible. When fraud is committed, we all lose.

A Practical Guide for Educators

The only viable path forward is to shift our mindset from policing to integration, adapting our methods to leverage AI’s strengths while mitigating its weaknesses.

Redesigning Assignments for the AI Era

With the traditional take-home essay now vulnerable to automation, educators must redesign assignments to incorporate AI as a tool for thinking, not a machine for answers. This requires a fundamental shift in what we choose to assess.

The most effective strategy is to focus on process, not just product. Instead of grading only the final essay or report, the assessment can be expanded to include the student’s engagement with the AI. Requiring students to submit their chat logs or a written reflection on their process—detailing the prompts they used, how they evaluated the AI’s output, and the modifications they made—makes their thinking visible. This turns the inquiry itself into the gradable artifact, rewarding critical engagement over simple content generation.

Another powerful approach is to turn students into AI critics. Instead of asking them to produce a text, assign them the task of deconstructing an AI-generated one. For example, a student could be asked to prompt an AI to write an essay on a historical event and then write their own analysis of its factual errors, logical fallacies, and underlying biases. This transforms the assignment from a simple writing task into a high-level critical thinking exercise, teaching students to be skeptical and analytical consumers of AI-generated content.

Finally, it is essential to emphasize human-centric assessments that are inherently resistant to automation. These methods evaluate skills that AI cannot replicate, such as real-time argumentation, interpersonal collaboration, and embodied knowledge. This includes a renewed focus on in-class discussions and Socratic seminars, oral exams and presentations, timed hand-written essays, and hands-on lab work or collaborative projects. While these redesigned assignments require a different kind of engagement, the time saved by using AI for administrative tasks can be reinvested here, creating a more sustainable and pedagogically valuable workflow.

AI as a Teacher’s Super-Assistant

AI’s greatest potential may lie in its ability to reduce the significant administrative burden on teachers, freeing them up to focus on the deeply human work of teaching and mentoring.

As a tool for lesson planning and differentiation, AI can be an invaluable creative partner. An educator can brainstorm engaging lesson plans, get suggestions for creative activities, or generate differentiated materials—such as simplified texts or vocabulary lists—for students with diverse learning needs in a fraction of the time it would take manually. For instance, a teacher could use a prompt like: “Act as an instructional designer. Create a 45-minute lesson plan for 10th graders on the causes of World War I, including a hook, a collaborative activity, and a formative assessment.”

For rubric and feedback generation, AI can be truly transformative. It can draft clear, comprehensive grading rubrics in seconds. More importantly, it can help solve the feedback bottleneck by providing initial, personalized feedback on student work. An educator can quickly review a student’s draft, identify key areas for improvement, and instruct the AI to provide detailed, constructive feedback on those specific points, without rewriting the text for the student. The teacher then reviews and approves the AI’s feedback before sending it. This “human-in-the-loop” model allows teachers to provide timely, detailed, and individualized feedback at a scale that was previously impossible. A teacher might use a prompt like: “Here is a paragraph I wrote. Provide feedback focusing on the strength of their topic sentence and their use of evidence, but do not rewrite it for them.”

Fostering an AI-Ready Classroom

Creating a healthy learning environment in the age of AI requires a proactive approach centered on clear policies, digital literacy, and open communication.

The foundation is to establish a clear classroom AI policy. Every educator should develop a simple, flexible policy for AI use and review it regularly. This policy should function as a guide for ethical engagement, not a list of prohibitions. It is crucial to define what constitutes constructive, ethical use (e.g., brainstorming, getting feedback on one’s own writing) versus what constitutes academic dishonesty (e.g., submitting AI-generated text as one’s own).

Beyond rules, educators must integrate AI literacy into the curriculum. It cannot be assumed that students understand how these tools work. This means dedicating class time to educating students on the capabilities, limitations, and ethical considerations of AI. This includes teaching practical skills like effective prompt engineering and essential concepts like how to spot AI “hallucinations” and the subtle ways that training data can introduce bias into the model’s output.

A simple and effective way to guide students is to create and share custom prompts and reusable AIs. By crafting prompts that are tailored to specific pedagogical goals—for example, a template designed to encourage critical analysis of a source—educators can model effective AI use. An even more powerful extension of this is to create shareable, custom AIs, often called “Custom GPTs” or “Gems.” These are specialized versions of the AI that are pre-loaded with specific instructions and context. An educator could create a “History Thesis Helper” that is an expert in their course material, or a “Lab Report Formatter” that guides students through the required structure. Sharing these resources not only helps students get better results but also embeds the desired learning process directly into the tool they are using.

Finally, it is vital to foster open dialogue. An educator should create a classroom culture where students feel comfortable and safe discussing the role of AI in their learning, asking questions, and even sharing their mistakes. By addressing the ethical implications and potential pitfalls of AI tools openly, the classroom becomes a collaborative space for exploring this new technology, fostering a sense of shared responsibility for its ethical use.

It is important to recognize that “AI burnout” is a reality. Many educators feel an immense pressure to adapt to everything at once, and that they have no time to do so. But this is not true. While we cannot dismiss AI, we do not have to change everything at the same time. The most sustainable path is one of small, deliberate experiments. By injecting AI into the easier parts of our teaching tasks first, we can achieve some easy wins, build our confidence, and give ourselves the time to reflect on the consequences before moving on to more ambitious integrations. The checklist below offers a simple way to begin.

A Four-Step Checklist for Educators

For educators feeling overwhelmed, here is a simple, actionable checklist to begin integrating AI into your practice:

1. Create and Discuss Your AI Policy: Draft your classroom AI policy using the appendix as a model. The most important step is to discuss it openly with your students on the first day. Frame it as a shared agreement for ethical engagement.

2. Use AI as an Assistant for One Task: Pick one administrative task this week and use an AI to help. Draft a lesson plan, create a rubric for an upcoming assignment, or generate a set of discussion questions. Experience the tool’s power and limitations firsthand.

3. Redesign One Assignment: Choose one of your existing assignments and brainstorm how you could redesign it to focus more on process, critical evaluation, or in-class performance. Start small and iterate.

4. Share a Resource: Create and share a custom GPT or a well-crafted prompt template designed to help your students kickstart one self-study activity or assignment. This models good practice and provides a valuable resource.

A Guide for the Modern Learner

For students, AI can be the most powerful learning tool ever created, but only if used with intention and integrity. The goal is to use AI to learn, not to short-circuit your own understanding. This requires a conscious shift from viewing AI as an answer machine to viewing it as a thinking partner.

Your Responsibilities as a User

Ethical use of AI begins with a clear understanding of your responsibilities. First and foremost, you must verify and clarify policies. Every course and institution will have different guidelines for AI use; it is your responsibility to know them and, when in doubt, to ask your instructor. Second, practice transparent disclosure. Being honest about how and where you have used AI in your assignments is a cornerstone of academic integrity and builds trust with your educators. Finally, you must protect sensitive information. Never input personal, confidential, or proprietary data into public AI models, as you have no control over how that data might be used or stored.

Using AI to Kickstart Your Work

One of the most effective and ethical ways to use AI is as a brainstorming partner to overcome the inertia of a blank page. You can use AI to generate initial ideas for a project, create a structured outline for an essay, or synthesize the key points from a long article. In this role, the AI acts as a catalyst for your own thinking, providing a foundation upon which you can build your original work. The goal is to use it to support your thinking, not replace it.

Using AI to Deepen Understanding

Instead of asking for a direct answer, use AI to guide you toward your own understanding. You can turn the AI into a Socratic partner that asks you questions instead of giving you solutions. For example, a prompt like “I’m trying to understand the causes of the French Revolution. Don’t list them for me. Instead, ask me questions that will lead me to the key factors” transforms a passive query into an active learning exercise. This approach reintroduces the “desirable difficulty” that is essential for true learning, using the AI to guide you rather than carry you.

AI is also an excellent tool for concept exploration. When faced with a complex idea, you can ask the AI to explain it in simpler terms or through an analogy, such as “Explain the concept of general relativity to me as if I were 12 years old.” This helps you build an intuitive grasp of the material that goes beyond rote memorization.

Using AI to Refine Your Skills

AI can be an invaluable coach for improving your practical skills through iterative feedback. As a writing coach, it can offer suggestions on clarity, tone, and structure without doing the writing for you. You can submit a paragraph you have written and ask for specific feedback, such as “Can you suggest three stronger verbs I could use in this sentence?”

As a practice partner, AI can generate an infinite number of practice problems for subjects like math, coding, or language vocabulary. You can ask it to create a quiz for you and then, crucially, to provide detailed explanations for any questions you get wrong, allowing you to learn from your mistakes in a low-stakes environment.

Build Your Own AI Tools

Beyond one-off prompts, the next level of AI literacy is learning to create your own reusable AI assistants. Modern AI platforms allow you to create “Custom GPTs” or “Gems”—specialized versions of the AI that you pre-program with your own instructions and knowledge. This is a powerful way to personalize your learning. For example, you could build a “Study Buddy” and upload all your course notes, empowering it to quiz you on the specific material. You could create a “Socratic Tutor” that is permanently instructed to only ask you guiding questions and never give direct answers. By building your own tools, you move from being a simple user to a creator, a skill that is becoming increasingly valuable.

Developing AI Literacy

Ultimately, the most important skill for a 21st-century learner is not just knowing how to use AI, but knowing how to critically evaluate its output. Never trust blindly. This new “AI literacy” is built on three pillars.

First, always be skeptical. Treat every statement an AI generates as a claim, not a fact. Second, fact-check everything. AI models can and will “hallucinate” incorrect information with complete confidence. You are the ultimate authority and are responsible for the accuracy of your work. Always use trusted, primary sources to verify any factual information the AI provides. Finally, learn to look for bias. Understand that the AI’s training data is a reflection of the vast and messy internet, full of human biases and stereotypes. Always question the perspective of the text it generates and be aware of its inherent limitations.

Putting It All Together

Here is a step-by-step example of how you might ethically use AI to help with a research paper:

1. Brainstorming: Use the AI to explore potential topics and narrow your focus.

2. Outlining: Work with the AI to structure your main arguments and create a logical outline.

3. Research: Use the AI to find sources or summarize articles, but always go to the original source to read it yourself and fact-check every claim.

4. Drafting: Write the full draft in your own words, using your outline and research.

5. Feedback: Ask the AI for feedback on the clarity, structure, and style of your draft.

6. Submission Checklist: Before submitting, review this list:

   • Have I fact-checked every claim that originated from the AI?

   • Can I explain and defend every part of this work in my own words?

   • Have I followed my instructor’s AI policy to the letter?

   • Does my declaration accurately and specifically describe how I used AI in this assignment?

Conclusion

The techno-pragmatist ethos that guides this book is rooted in a fundamental belief: the future is not predetermined. Technology is a tool whose impact is profoundly shaped by how we choose to employ it, and this is nowhere more true than in education. As a college professor, this is not an abstract debate for me; it is a topic I care about deeply, and I feel a profound responsibility to get it right.

The challenge is not to resist this new technology, but to harness it with wisdom. Instead of chasing the flawed ideal of automation or descending into an adversarial relationship based on detection, we must embrace a necessary pedagogical shift. The central problem in modern education is not a lack of content, but a scarcity of timely, personalized feedback. High student-to-teacher ratios make it nearly impossible for educators to provide the deep, iterative guidance that is crucial for student growth.

This is where AI can create a true revolution. Therefore, the true north for AI in education is not automation, but augmentation. We must leverage AI to solve the feedback bottleneck, using it to do what it does best—process information and provide feedback at scale—so that we, educators and learners, can focus on what we do best: questioning, creating, and collaborating within a human-centered community.

It is from this techno-pragmatist perspective that we have offered these guides. The strategies herein are not just tips and tricks; they are a framework for shouldering the shared responsibility of building a new AI literacy, ensuring that these powerful tools serve, rather than subvert, the timeless goals of a meaningful education.

Thanks for reading so far! As a complementary resource, here is a draft of an AI Policy for STEM classes. Feel free to, share, modify, and reuse it as you see fit.

Example AI Policy for STEM Classrooms

31.6KB ∙ PDF file

Download

Download

This is a first draft of Chapter 7 of my upcoming book Mostly Harmless AI. Please, do share with me all your comments, suggestions, criticisms, and ideas.

Leave a comment

This article is based on Chapter 4 of my upcoming book Mostly Harmless AI.

Having journeyed through the foundations of artificial intelligence—its history, its mechanics, and its limitations—we arrive at the most pressing question: How do we actually use the powerful new tools it has produced?

This article serves as a bridge from the theoretical to the practical. This is not a list of traditional prompt engineering hacks. The internet is filled with tips and tricks for coaxing a specific output from a language model for a single task. Instead, the advice that follows builds on our foundational understanding to offer something more durable: a general mindset for working with these tools. This approach is about engaging with language models in a way that allows you to get the best out of them without subcontracting your own critical thinking. It is a methodology for augmenting your intellect, not replacing it.

The principles you learn here are foundational, offering a universal toolkit for interacting with large language models in daily life—whether you are planning a vacation, trying to understand a complex news article, or drafting a simple email. In the following chapters (in the book), we explore how to adapt and intensify these practices for specialized, high-stakes professional environments. But first, every user must learn how to engage with these powerful yet fallible tools safely, critically, and effectively.

A Methodology for Effective Interactions with Language Models

To move beyond simple queries and unlock the true potential of language models, we need a more structured approach. This methodology is divided into three parts: establishing the right Mindset, employing effective Tactics during the conversation, and building a System to make your successes repeatable.

The Mindset

The most significant shift is in your mental model. Instead of treating the model as a search engine, you should approach it as a conversational partner. This means recognizing that the interaction is iterative and that your most important role is to guide the conversation.

A key part of this mindset is adopting a Socratic (or inquiring) approach, where you use the model not just to get answers, but to help you ask better questions. This is invaluable for sensitive and important tasks.

For example, instead of starting with “Write an email asking for a raise,” a partner-based approach would be to ask the model to guide you: “I need to write an email to my manager to ask for a raise. What are the key pieces of information and evidence I should gather first to make the strongest possible case?” The model will then prompt you for your accomplishments and market data, helping you build your argument before a single word is written.

Similarly, when organizing a child’s birthday party, you could ask, “I’m planning a science-themed party for my 7-year-old. What are the key logistical details I need to consider to make sure it runs smoothly?” In both cases, you are using the model to help you define the problem, which is a far more powerful use of its capabilities.

The Tactics

With the right mindset, you can employ specific tactics to steer the conversation toward a high-quality outcome. The most fundamental tactic is to be explicit and strategic with your queries. To ground the model’s response in reliable information, tell it where to look.

A generic query for medical advice is risky, whereas a much safer prompt would be: “Search for information from the Mayo Clinic and the World Health Organization on the common symptoms of iron deficiency.” This specificity is also crucial when comparing complex options, like, for example, buying an EV, you can be as specific as: “Compare the Tesla Model 3, the Hyundai Ioniq 5, and the Ford Mustang Mach-E for a family of four. Focus on real-world range, charging speed on a standard home charger, and available cargo space.”

To get an even more robust answer, you can move beyond a single query and assemble a ‘committee of experts.’ A single language model will give you its most statistically likely answer, which might not be the most creative or well-rounded one. To overcome this, you can generate multiple, independent perspectives.

For the EV comparison, you could open three separate conversation windows. In the first, you’d ask the model to act as a pragmatic engineer and, perhaps, the model will argue for the Hyundai. In the second prompt, you’d ask it to be a tech enthusiast, and maybe it makes the case for the Tesla. In the third, you’d have it act as a family-focused reviewer, causing it to arguing for the Ford.

By copying these three independent analyses into a final chat window, you can then ask the model to act as a senior editor, synthesizing the competing viewpoints into a final, balanced recommendation that weighs factors like cost, range, and reliability.

Finally, after the model provides a response—either a single answer or a synthesized one from your committee—you can employ self-criticism as a final refinement tactic.

Once you have a draft of your email asking for a raise, you can prompt it: “Read the email you just drafted. Now, act as my manager who is busy and skeptical. What parts of this email are unconvincing? Is the tone too demanding or not confident enough?” This critical step often surfaces weaknesses that you might have missed, allowing you to create a much stronger final product.

The System

The final part of the methodology is to turn your successful interactions into a repeatable system. A common mistake is to treat prompts as disposable. A more powerful approach is to build a library of reusable prompts, thinking of them as personal “natural language programs.” The multi-step process you used to plan the birthday party can be saved as a “Kids’ Party Planner” template.

The Socratic prompt that helped you prepare for your salary negotiation can be generalized into a “Career Conversation Prep” tool. The ultimate expression of this principle is the use of features like OpenAI’s “Custom GPTs,” which allow you to encapsulate a complex task into a dedicated tool that you or your team can use with a simple request.

A Practical Example

To see how these principles combine into a powerful workflow, let’s walk through a comprehensive, real-world task: planning a 10-day family vacation to Italy.

Rather than beginning with a vague request like “plan a trip,” the process starts by applying the Socratic approach. You would first ask the model to frame the problem for you: “I want to plan a 10-day family vacation to Italy. What key information do you need from me to create the best possible itinerary?”

This immediately shifts the dynamic, positioning the model as a guided partner. In response, it would act as a consultant, asking for crucial details like the number of travelers, the children’s ages, your budget, family interests, and preferred travel pace.

Once you’ve provided this context, the next step is to ensure alignment. You would instruct the model to synthesize and confirm the constraints: “Great, thank you. Based on my answers, please summarize all of my constraints for this trip in a structured list.”

With a clear, confirmed set of requirements, you can then confidently ask for a first draft. The iterative heart of the process begins now. Upon receiving the initial itinerary, you would employ the self-criticism tactic: “This is a good start. Now, act as a skeptical travel agent. Criticize this itinerary and tell me what’s missing or what could go wrong.”

The model might point out that visiting three major cities in ten days is too ambitious for a family with young children. Based on this valuable feedback, you can guide the revision, continuing this loop of drafting and critiquing until the plan is refined to your satisfaction. Only then would you ask for the final, detailed output.

The final, powerful step is to generalize this success. You would ask the model to convert the entire conversation into a reusable “Family Vacation Planner” template, complete with placeholders for key details. This turns a one-time effort into a valuable, programmable asset for future trips, demonstrating the true power of thinking of prompts as reusable programs.

Common Pitfalls for the Everyday User

The good practices above are designed to improve the quality of a language model’s output. This section focuses on the mental traps and risks you must be aware of to use these tools safely.

The “Eliza Effect” and Misplaced Trust

Because chatbots are designed to be conversational and helpful, it’s easy to start treating them as if they have genuine understanding, intentions, or even consciousness. This is a modern version of the “ELIZA effect” we discussed in the history chapter. The danger is that this leads to misplaced trust, where we stop critically questioning the model’s output because it feels so confident and knowledgeable. This is the psychological trap that makes us vulnerable to hallucinations; we are less likely to fact-check a “partner” than a machine.

Cognitive Offloading and The “Lazy Brain” Problem

The ease of asking a language model to summarize an article, draft an email, or brainstorm ideas can lead to a subtle but significant danger: cognitive offloading. By outsourcing the fundamental work of thinking, synthesizing, and structuring our thoughts, we risk letting our own critical thinking and creative muscles atrophy. The goal is to use these tools to think better, not to think less. Over-reliance can make us less capable problem-solvers in the long run.

The Privacy Risk of Casual Conversation

In a casual conversation with a chatbot, it’s easy to forget that you are interacting with a complex system run by a corporation. Users often paste sensitive personal information—medical details, financial data, private emails, proprietary work content—into public language models without considering where that data goes, how it’s used for future training, or who might have access to it. What you tell the model does not stay between you and the model.

You Are the Final Authority

The techniques above teach you how to get better raw material from the language model. This final principle is about what you, the human, must do with that material. It is the most critical step in using these tools responsibly.

First, never trust, always verify. The language model is an unreliable narrator. Treat its output as a well-written first draft, not a finished fact. For any critical piece of information—a date, a statistic, a medical suggestion, a legal point—you must verify it using an independent, authoritative source. The model can help you find potential sources, but you are the fact-checker.

Second, synthesize, don’t just copy-paste. The model’s output is information; your goal is knowledge. The most important work happens after the model has responded. Your job is to synthesize its suggestions with your own experience, judgment, and goals. The model can generate a list of tourist sites for your Italy trip, but only you can synthesize that into a vacation plan that feels right for your family.

Finally, own the outcome. The language model is a tool, and you are the user. Any decision made, any email sent, or any action taken based on the model’s output is your responsibility. This principle of accountability is non-negotiable. The model is an assistant that can help you think, but it is not a replacement for your personal judgment.

Conclusion

The journey from a novice user to a skilled one is not about memorizing clever prompts; it’s about a fundamental shift in mindset. Instead of treating generative AI as a vending machine for answers—an approach fraught with risks of shallowness, bias, and error—we’ve seen the power of engaging it as a conversational partner.

The practices outlined in this article—the Socratic method, strategic querying, and, most importantly, critical verification—form a framework for responsible engagement. This framework places you, the user, firmly in the driver’s seat.

The quality of the model’s output is not a feature of the model alone; it is a direct reflection of the quality of your guidance and the rigor of your review. You are not just a prompter; you are a director, a critic, and a synthesizer. This is what makes these powerful tools ‘mostly harmless’: not their inherent nature, but our commitment to using them with critical awareness and human authority.

By mastering these foundational skills, you are not just learning to use a new tool. You are developing a new form of literacy for the 21st century. As we move into the specialized applications for knowledge workers, developers, and creatives in the following chapters (of the book), this ability to think with AI, not just ask of it, will be your most valuable asset.

Thanks again for reading. If you want to dive deeper into Artificial Intelligence and learn to make the best out of it, from a techno-pragmatist, human-centered, responsible perspective, please check out my book Mostly Harmless AI.

This article is based on Chapter 5 of my in-progress book Mostly Harmless AI.

Few developments in the generative AI space have been as exciting lately as the rise of code generators. The evolution of these AI coding assistants is best understood not as a single leap, but as a progression of capabilities, moving from simple autocomplete to what may one day be fully autonomous agents.

At their core, code generators are Large Language Models trained on vast amounts of public code. They treat programming languages just like human languages, learning the patterns, syntax, and structure to predict what comes next. These models can take a natural language prompt and some contextual code and produce new code that mostly aligns with the prompt's intention. For example, you can provide a function signature and a comment like, "This function finds the minimum of an unsorted list," and the model will generate the function's body.

This uncanny ability to comprehend and generate code based on human communication is transforming the development landscape, but it also requires special considerations, as code is not just another natural language.

In this article, we will explore the landscape of AI for software development. We will begin by looking under the hood, examining the spectrum of capabilities that allow AI to generate code. Next, we will explore the use cases for developers across the development lifecycle. Then, we will discuss some important things to keep in mind, from hallucinations and security to theoretical limitations of AI for coding. Finally, we will look to the future of coding, consider how the developer role is evolving, and try to answer one crucial question: is coding dead?

How to Make a Code Generator

Let's imagine we are building our own code generator from scratch. The journey from a simple code predictor to a sophisticated development partner is a journey of adding layers of capability, moving up a spectrum of increasing autonomy.

The first thing we want is next-token prediction for code. The foundational layer is built on unsupervised training, making it essentially autocomplete on steroids, like a super duper IntelliSense. We start by training a model on vast amounts of code, teaching it to predict the most likely next token based on the immediate context. If we have variables or functions declared nearby, our model is more likely to generate code that references them, simply because that's the most common pattern in its training data.

Now that we have a basic generator, our next step is to teach it to follow instructions. To do this, we can compile a dataset of instruction pairs—for example, a natural language command like, "In the previous code, change the loop to be more efficient," paired with the corrected code. By training on these examples, our model learns to go beyond simple prediction and follow specific, human-given directions. We can further enhance this process with Reinforcement Learning, where we have human or automatic evaluators rank different code outputs. This teaches our model to not only generate syntactically correct code, but also to respect desired styles and naming conventions.

Our generator is getting smarter, but it's still limited by the immediate context. To give it a long-term memory, we move to context-aware generation. We can dramatically enhance our model's ability to generate relevant code by allowing it to pull from a broader context. This is a form of Retrieval-Augmented Generation (RAG) for Code. We can index an entire codebase or external API documentation, allowing our model to find relevant examples and patterns. When a developer asks a question, our system retrieves these examples and feeds them into the prompt, allowing the model to generate accurate code by combining and refactoring snippets from the provided context, even for libraries it wasn't explicitly trained on.

So far, our model can only write code. To up our game, we can give it the ability to interact with its environment by making it use external tools. This represents a significant leap. We can equip our model with a set of tools it can invoke on demand. For example, in response to a prompt like "add a library for charting," our model could invoke a tool to install the missing dependency in the project. If you ask it to "check whether this works," it could invoke another tool to run the unit tests and report back the results. It could even use tools to directly modify files in the codebase. By giving our model the ability to take actions beyond just generating text, we empower it to participate more actively in the development process.

The final step on our journey is to give our code generator a bit of autonomy, creating what’s called an agentic system. This is the most advanced and forward-looking form of AI-based code generation. We can design an AI agent that takes a high-level goal, breaks it down into sub-tasks, writes code, generates tests, runs the code, and then analyzes the output or errors. Based on the results, it can then debug or modify the source code in a continuous loop, acting as a semi-autonomous developer to see a task through from start to finish.

Beyond these training methodologies, we can leverage the formal nature of code to improve our model's performance. Unlike natural language, code has strict syntactic rules that can be programmatically checked. One simple but effective technique is trial and error during inference: we can have our model produce several potential code snippets, run them through a linter, and automatically reject any that have parsing errors.

More advanced techniques can pre-process the training data, for instance by normalizing all variable names to a generic format like var0, var1, etc. This makes it much easier for our model to learn the structural relationships in code without being distracted by specific naming conventions, and we can substitute the actual names back in a post-processing step. These tricks leverage the fact that we are dealing with a very restricted syntax to make it easier for our language model to learn the rules.

Finally, to create the ultimate specialized assistant, we can go beyond RAG and fine-tune a model on a specific codebase. While RAG provides external context, fine-tuning actually updates the model's internal weights. By training a model on a company's entire private and proprietary codebase, we can create a version that has deeply internalized that organization's specific architectural patterns, internal APIs, and coding standards. This results in an AI partner that not only answers questions correctly but does so in a way that is idiomatic and aligned with the team established practices.

Use Cases for Developers

Understanding the engine is one thing; knowing how to use it is another. AI offers you a powerful toolbox that you can apply across the entire software development lifecycle. In this section, you will explore practical use cases across three key phases: Ideation and Design, Implementation and Development, and Verification and Explanation. You will see how you can use both sophisticated, LLM-based coding tools integrated directly into your IDE, as well as techniques that you can use with standard, general-purpose chat apps like ChatGPT or Claude, requiring no special integration at all.

Phase 1: Ideation and Design

Before you even dare writing a single line of AI-generated code, you can already use LLMs as powerful brainstorming partners for exploration and design. This is one of the most accessible ways you can use AI, as it doesn't require a specific tool or editor extension; you can do it effectively using general-purpose conversational AI applications like ChatGPT, Perplexity, or Gemini. Models with live browsing capabilities are often even better for this phase, as they can pull in the latest information about new frameworks, libraries, and design patterns.

The key is for you to treat this phase as an interactive exploration. Instead of asking for a single, ready-made answer, you should guide the model through an ideation process. Here's how you can do it: use a chain-of-thought approach by asking the model not just for a solution, but to "think step-by-step" through the pros and cons of different architectural choices.

A powerful pattern you can use is to ask the model to generate several variants—for example, "Propose three different ways to design the database schema for a social media app." Then, you can discuss the options back and forth, using self-critique prompts to have the model compare the alternatives it just generated. At the end of this collaborative session, you can ask the model to provide a structured summary of all the design decisions you have agreed upon, acting as an executive design document.

With this document in hand, you can then start a new, more focused session with a code-oriented model for the actual implementation.

Phase 2: Implementation and Development

The most straightforward way you can use AI in this phase is for generating short, self-contained code snippets. This can be for a well-known algorithm, a common pattern, or the use of a well-documented API. This is a task you can accomplish with any standard chat app, even outside your IDE. This is especially powerful for navigating the complex world of APIs and libraries.

As a professional programmer, you probably aren't spending that much time doing basic coding, like inserting numbers in a list. No, reality is 90% of the code you write is interface code with some external library you may not know well. Instead of manually searching documentation, you can simply ask an AI assistant, "How do I use this library to make a query that does X?" and get a ready-to-use snippet.

The next level of integration is bringing AI directly into your IDE. This can start with simple code completion, but the real power comes from integrating a full chat experience. This allows you to highlight a block of code and ask for specific changes, such as, "Refactor this function to be more efficient," and have the model modify the file directly. If the model has RAG capabilities and can scan your entire codebase, it gets even better. The modifications and additions it suggests will be consistent with your existing coding style and use your own libraries and methods, making the integration seamless.

At the far end of the spectrum is the full agentic mode, which is still in its infancy with tools like Cursor. This offers a much more hands-off development experience. Here, you can give a coding agent a high-level task, and it can modify several files, create new ones, and even run commands in the terminal to install missing dependencies.

Finally, you don't always need a full IDE. For one-off scripts or quick prototypes, you can use the "Canvas mode" in apps like ChatGPT, Claude, or Gemini. These provide a simple editor-like interface where you can iterate back and forth with the model to update a script. Some tools even allow you to run these scripts directly in the cloud, letting you build and test disposable web apps instantly.

Working with these tools introduces a new core skill, an AI-in-the-loop coding workflow—the day-to-day interactive process of collaborating with an AI. It involves an iterative cycle of prompting with a clear goal, carefully reviewing the AI's output, correcting its mistakes or flawed assumptions, and then re-prompting with more specific instructions or feedback.

Phase 3: Verification and Explanation

To ensure your code quality, you can use an AI to help generate a wide range of test cases. This is especially useful for uncovering corner cases that might not be immediately obvious, such as handling empty inputs, maximum values, or unusual user behaviors. You can do this with integrated AI coding tools, but it can also be as easy as uploading your codebase or relevant files to a standard chat app and asking it to suggest test cases.

You can ask for both code-based tests (like unit and integration tests) as well as descriptive tests (like user stories or manual testing scripts). In all these scenarios, it helps to instruct the model with a Chain-of-Thought prompt, asking it to first explain what behavior it wants to test, and only then provide the actual test. This ensures the tests are intentional and well-understood.

Furthermore, when you're faced with a cryptic error, you can use AI for debugging. You can feed the AI the error message, stack trace, and relevant code, and it can analyze the context to suggest potential causes for the bug and possible fixes, acting as an experienced pair programmer.

The opposite, code-to-language direction allows you to create powerful new workflows for understanding code. You can ask an AI for automatic documentation of functions or for natural language explanations of a complex code fragment. This can be done directly inside your IDE with an integrated tool, or with standard chat apps. For example, some tools allow you to connect a public GitHub repository and ask high-level questions on the fly, which is very good for getting a quick overview of a new codebase.

A particularly valuable use case is in legacy code modernization. One of the biggest challenges in the software industry is maintaining and updating old codebases. You can use AI to tackle this problem by feeding it legacy code (e.g., from an old COBOL or Java system) and asking it to analyze the logic, add explanatory comments, or even translate the entire system to a modern language and architecture. This can dramatically reduce the cost and risk associated with modernizing critical systems.

However, you should be aware of the critical gap between syntax and semantics—that is, between understanding what the code says versus what the code does.

The weaker models are mostly limited to describing what the code is saying syntactically (e.g., "this variable is changed to this array position"), this capability is improving all the time. More powerful models can often provide higher-level, semantic explanations of what the code is doing (e.g., "this loop is ensuring the first part of the array is always sorted"). But even the best models may not be able to grasp the full architectural details or business logic of a complex application.

Putting It All Together

Putting this all together, let's see how a complete workflow might look for tackling a specific, somewhat complicated feature in an ongoing app, like adding OAuth login.

First, you would start in ideation mode, interacting mostly in text with the AI. You would discuss a high-level overview of the required architecture changes, which parts of the app might be impacted, and the best libraries to use. The goal here is to produce a clear design roadmap before any code is written.

Next, you would move to implementation mode, going full hands-on with an agentic tool. You could assign the agent the high-level task from your roadmap: "Implement the OAuth login feature using the chosen library." The agent would then get to work, creating new files, modifying existing ones, and writing the necessary code. As it encounters errors or ambiguities, you would engage in a back-and-forth conversation to guide it, but the bulk of the mechanical coding would be handled by the agent.

Finally, you would enter review mode. Once the agent reports that the feature is complete, you could have a final conversation with the AI. You could ask it to analyze the git diff of all the changes it made, explain the rationale for its implementation choices, and generate comprehensive documentation for a pull request. After your final review and approval, you would then submit the PR for human review by your team.

Things to Keep in Mind

While the toolbox is powerful, it comes with sharp edges. The most important limitation in language modeling, in general, has been called the problem of hallucinations. In the context of code, this means AI-generated code is not infallible and can contain subtle bugs that require constant vigilance.

Hallucinations and Mistakes

The simplest way you can see hallucinations is when you get code that uses a new variable that doesn't exist or fails to close a parenthesis. Unlike with natural language, you can often detect these syntactic errors automatically with a linter or compiler, so many of the more harmless hallucinations are not relevant as they won't introduce subtle bugs.

A slightly more difficult hallucination is what we can call a semantic hallucination, where the model uses a wrong variable or function name that does exist in your codebase. In this case, you will not get a compiler error because you're using an existing symbol, but you will get the wrong behavior. This is much harder to find because it has the same problem as most hallucinations: you have to review the code and be knowledgeable enough to have been able to generate that code yourself.

The most insidious errors are logical flaws. This occurs when the code doesn't do anything obviously wrong—it uses the right variables and looks plausible—but it has some subtle logical mistake that leads to a bug. For example, finding that a variable is not updated at the right moment in a nested loop is a tricky problem even for human experts. These kinds of mistakes will introduce subtle, hard-to-detect bugs.

But even if the bugs are no worse than what a human would introduce, they pose a threat because of "automation bias." When you check code written by humans, you expect bugs. But when you're looking at machine-generated code, the only way programmers have ever interacted with it has been with rule-based systems like compilers, and that code is basically without mistakes.

So even if the language model makes errors that are, on average, no worse than what a regular programmer would make, they can still be harder to detect because they won't be the exact same mistakes a human would make, and we may be less on guard.

AI's Impact on Technical Debt

The rapid generation of code by AI presents a double-edged sword for technical debt. On one hand, you can use AI as a powerful tool to reduce existing debt. You can ask it to analyze your codebase for inefficiencies, suggest refactorings, or add missing documentation and tests, thereby improving code quality.

On the other hand, the very speed of AI can create new and more complex forms of technical debt. Relying heavily on "vibe coding" to quickly generate features without rigorous human review can lead to a codebase filled with poorly understood, inefficient, or subtly buggy logic. This AI-generated debt can be even harder to untangle later, as the original human intent behind the high-level prompt may be lost.

Biases in Generated Solutions

Models trained on a vast corpus of public code from the internet will inevitably learn from outdated examples. This can lead them to perpetuate outdated practices by suggesting deprecated functions, old library versions, or inefficient algorithms that are no longer considered best practice. An AI model will also often default to the most statistically common solution it has seen in its training data.

This can stifle creativity and lead to a homogenization of code, discouraging the exploration of more elegant or contextually appropriate solutions. Finally, just as AI can perpetuate harmful societal biases, it can also reproduce human biases from the code it was trained on. This can manifest as non-inclusive language in generated comments or variable names.

Security and Licensing Risks

A significant risk is that an AI can generate code with known security vulnerabilities. If the model was trained on public code containing flaws like SQL injection or buffer overflows, it may reproduce those same insecure patterns in its suggestions, creating a major security risk for the application.

Furthermore, the use of AI-generated code introduces complex legal questions. A model might reproduce a code snippet verbatim from a repository with a restrictive open-source license (like the GPL), inadvertently pulling that license's requirements into a proprietary project. The legal ownership of the AI-generated code itself remains a gray area, creating potential intellectual property challenges for companies.

The Economics of AI Development Tools

While these AI tools offer significant productivity boosts, they are not free. For development teams and organizations, it's important to consider the practical economics of their adoption. Most advanced AI coding assistants operate on a subscription model, which introduces a new operational cost. Team leads and CTOs must perform a cost-benefit analysis, weighing the price of the tools against the expected gains in developer speed, code quality, and reduced time-to-market. The return on investment (ROI) will depend heavily on how well a team integrates these tools into their workflow and whether the productivity gains justify the recurring expense.

Theoretical Limitations

Beyond the practical issues of hallucinations and biases, there is a more fundamental, formal limitation to what we can do automatically. This is captured by Rice's theorem, a cornerstone of theoretical computer science. In short, the theorem proves that there is no algorithm that can automatically check for any non-trivial semantic property of a program.

What does this mean in practice? A "non-trivial semantic property" is basically any interesting question about what a program does. For example: "Does this program ever crash?" or "Will this function always return a positive number?" or "Is this code free of security vulnerabilities?" Rice's theorem tells us that it is mathematically impossible to build a universal program that can answer these kinds of questions for every possible piece of code.

This highlights the theoretical impossibility of perfect, automated code verification. We will never be able to build an AI that can look at code generated by another AI (or a human) and formally guarantee that it does exactly what the natural language prompt intended. That problem is, in the general case, unsolvable.

However, this doesn't mean we should give up. Engineering isn't about theoretical perfection; it's about solving the average case in the best possible way and handling the most important edge cases reasonably well. While we can't achieve perfect verification, we can get pretty far with a combination of AI-generated tests, linters, and, most importantly, expert human review.

The Future of Coding

Given these tools and guardrails, the very nature of programming is set to transform. The focus will shift from the mechanics of writing code to the art of building systems. The term "vibe coding," popularized in developer communities, captures the essence of this shift. It describes a workflow where the developer's primary job is no longer to write precise, line-by-line syntax, but to describe the high-level behavior, intent, or "vibe" of the desired software to an AI partner. The focus moves from how to do something (the specific algorithm and syntax) to what needs to be done (the ultimate outcome and user experience), leaving the mechanical implementation details to the AI assistant.

This approach is incredibly powerful for rapid prototyping, hackathons, and short-term projects. A developer can quickly scaffold an entire application by describing its components in natural language, getting a functional prototype up and running in a fraction of the time it would take manually.

However, this method has significant limitations for larger, more detailed projects. "Vibe-based" instructions are often ambiguous and can be misinterpreted by the AI, leading to code that works for the happy path but fails on edge cases. For long-term, mission-critical software, the precision, maintainability, and strict adherence to architectural standards that come from deliberate, human-led coding remain indispensable. Vibe coding is a tool for speed and exploration, not a replacement for rigorous engineering.

In this new paradigm, future developers will become experts at wielding a suite of AI tools and agents. Skills in "prompt engineering," system design, and the critical review of AI output will become more valuable than the ability to recall specific syntax. The developer's role becomes one of guidance and orchestration, knowing which tool to use for which task and how to verify the results.

Looking ahead, this elevated role may involve assigning entire features or bug fixes to autonomous agents. These agents would manage the full lifecycle: understanding the ticket, writing the code, creating tests, committing to version control, and responding to feedback from the CI/CD pipeline. This doesn't eliminate the developer but elevates their role to that of a system architect and project manager, overseeing a team of AI agents.

Beyond the changes in workflow, it's worth contemplating how these tools will change the qualitative experience of being a developer. We must ask ourselves how it feels to code this way. Does offloading the cognitive burden of syntax and boilerplate make you dumber and cause you to forget how to code, or does it free up mental space, allowing you to become even more proficient in the things that truly matter—the high-level ideas and architecture?

We must also consider the social aspects. You now have a partner that is not a human. How will this impact teamwork? Will this AI partner become a virtual member of the team, participating in code reviews and design discussions? Or will it alienate developers into more lonely roles, as they interact more with their AI than with their human colleagues? How does a senior developer mentor a junior who can always get an instant answer from an AI, potentially masking gaps in their fundamental knowledge?

These are open questions we must navigate as we integrate these powerful new collaborators into our teams.

Final Remarks

So, Is Coding Dead?

There is a real concern that if AI can write 90% of the code in 10% of the time, nine out of ten programmers could be out of a job. And yes, every time automation has reached an existing industry, some jobs are destroyed as some skills become irrelevant.

However, I claim we must not fear the advent of AI coding assistants. Here’s why.

Writing code is by far neither the hardest nor the most time-consuming part of software development. The process of making software involves understanding requirements, talking with customers, user testing, and product design, all of which are at least one order of magnitude more difficult that actually typing code.

A hundredfold boost in productivity for a task that is only 10% of the overall process is huge, but it still leaves the other 90% of the human-centric work. We will still need to understand what our customers want, guide them through designing a software product, know the user base, and find a sustainable business model.

And no, you cannot simply simulate the end user with a language model, so the AI can prompt itself into making a usable product, because your end user will still be human. Human users are slow, get angry easily, don't understand your application, and don't know what it is they don't like about it. Until an AI can really replicate what it feels like to be a human—and at that point, will we still call it “artificial”—we can’t take the human out of the software development loop—or any creative loop, for the matter.

The biggest progress in the software creation process has always been because of innovation in the human side, not the machine side. Innovation in software engineering, management, and how you get people to work together and collaborate will continue to be the most important part of the software pipeline for a long time.

Furthermore, software is an industry that is nowhere near its saturation point. We have far more need for software than the number of people who can currently write it. Increased productivity will likely be met with increased demand, creating more and better software for more users.

Every leap in software productivity—from assembly to compilers, from C to object-oriented frameworks—has lowered the barrier to entry and brought more people into programming. AI tools will likely do the same, empowering more people to create software. The modern world runs on software, and in the future, basic programming literacy may become as common as basic math literacy is today.

Most people know enough math to get by in daily life without hiring a mathematician, and in the same way, more people will know enough programming to automate simple tasks. They will learn to say to their home computer, "When I get home, I want you to turn my lights on, but only if it's night and the electric bill is not above the average," and an AI will generate the code to make it happen. This expands the field rather than shrinking it.

So, should you learn to code? Definitely. There's going to be orders of magnitude more code written in the next few years than everything we've written in history.

But even if you never end up writing a single line of code unaided by AI—like I've never written a single line of production code unaided by syntax highlighting, a linter, or a type verifier—knowing how code works, how algorithms work, and why a specific programming construction works the way it works is the same as knowing basic math. Coding changes how your brain is wired, makes you think clearer, and increases your creativity.

Furthermore, even if you are not working in the software industry, learning to code is still an immensely enjoyable experience. Being able to create something that keeps working on its own is, I think, the ultimate toy.

So if you want to make a dent in the software industry and you're wondering if AI will get you out of the picture, don't worry. That won't happen anytime soon. Learn to code, learn the fundamentals, but also learn how to use these new tools. As in every moment in human history, if you apply yourself and do your best, you will be at the top of the league, and there will be a spot for you.

The following article is extracted from Chapter 1 of Mostly Harmless AI, a 160-page book on LLMs and everything you can, and cannot do with them. The book is in beta stage, meaning it has basically the final structure and content, but major revisions are still in progress. You can get it today at a 50% discount and ensure access to all future versions, including huge discounts on printed versions, as well as a community of like-minded readers.

What is a (Large) Language Model?

In machine learning, language modelling means guessing how likely a given sentence is.

For example, "the sun rises in the east and sets in the west" is a typical sentence with a high subjective probability. You would probably agree this is a sentence you’re likely to hear at least one. But a sentence with random words that don't mean anything has a low probability of ever being uttered by anyone, or written in a book.

Language modelling can be tricky because it's hard to say how likely a sentence is to “exist”. What does it even mean? In machine learning, we use a collection of texts called a corpus to help with this. Instead of the abstract, ontological question, we might ask something much more straightforward: How likely is it for this sentence to appear in all the written text, for example, on the internet?

However, if we only looked at sentences that already exist on the internet, language modelling wouldn't be very useful. We'd just say a sentence is either there or not, with a probability of 0 or 1. So instead, we can think about it in statistical terms like this: If the internet was made and erased many times, how often would this sentence show up?

To answer this question, we can think about whether a word will likely come after a group of words in a sentence. For example, "The sun rises in the east and sets in the..." What word would most likely come next? We want our language model to be able to guess that word.

Thus, we need to know how often a given word appears after a group of words. If we can do that, we can find the best word to complete the sentence. We keep doing this repeatedly to create sentences, conversations, and even full books.

Now, let's talk about the most common way to make this language modelling program work in practice. It's called statistical language modelling. We start with lots of text and learn how words correlate with other words. That is, we estimate the correlation of each word with a given context.

In simple terms, a context is a group of words around a specific word in a sentence. For example, in the sentence "the sun rises in the east and sets in the west," the word "east" is in the context of "{ the, sun, rises, and, sets }." If we look at many sentences, we can find words that are often in the same context. This helps us understand which words are related to each other.

For example, if we see "the capital of France is Paris" and "the capital of the United States is Washington," we can learn that Paris and France, as well as Washington and the United States, are related. They have the same relationship: being the capital of a country. We might not know what to call this relationship, but we can know it's the same type.

Statistical language modelling is thus making a model that can guess how often a word appears in a certain context, by using lots of data. This doesn't necessarily mean it truly understands the meaning of a sentence. But if we use enough data, it starts to look like the model can indeed capture at least some of the semantics.

The Simplest Language Model: N-grams

We've been building statistical language models since the early days of AI. The n-gram model is one of the simplest ones, storing the probability of each n-gram's occurrence. An n-gram is a collection of n words that appear together in common sentences. For example, in a 2-gram model, we count how many times pairs of words appear together in a large corpus, creating a table showing their frequency.

As we increase the n-grams to 3, 4, or 5, the collection of all n-grams becomes extremely large. Before the deep learning revolution, Google built a massive n-gram model from the entire internet with up to 5-grams. However, since the combination of all 5 words in English is huge, it only stored probabilities for the most common combinations.

This simple model counts words in a strict context when they're within a specific window size. It's very explicit, as each n-gram's probability or frequency is recorded. To compress this model further, we use embeddings.

Word Embeddings

An embedding is a mapping of some object—say, a word—to an n-dimensional vector of real numbers. Embeddings aim to transform semantic properties from an original space—i.e., words—into numerical properties of the embedding space. That is, we want words that occur together in context to map to similar vectors and form clusters in the embedding space.

Word2Vec, in 2011, was the first massively successful use of embeddings. Google trained a large embedding model using statistics from text all over the internet and discovered an amazing property: directions in the embedding space can encode semantic properties.

For instance, if you go take France and Paris, the vector needed to add to the word “France” to reach “Paris” is similar to the vector needed to add to the word “United States” to reach “Washington”. The semantic property has-capital was encoded as a specific direction in this space. Many other semantic properties were found encoded this way, too.

This was an early example of how encoding words in a dense vector space can capture some of their semantics.

Contextual Word Embeddings

The issue with Word2Vec is its assignment of a unique vector to each word, regardless of context. As words have different meanings in different contexts, many attempts were made to create contextual embeddings instead of static ones.

The most successful attempt is the transformer architecture, with BERT being the first example. The first transformer paper revolutionized natural language processing in artificial intelligence, providing a single tool to tackle various NLP problems.

A transformer is a neural network that generates a embedding of each word in the input text by considering the entire content of the input. This means each word's embedding changes according to its context. Additionally, a global embedding for an entire sentence, paragraph, or general fragment of text can be computed.

Why does this matter? Neural networks are among the most powerful machine learning paradigms. We can find embeddings for text, images, audio, categories, and programming code using a single representation. This enables machine learning across multiple domains using a consistent approach.

With neural networks, you can transform images to text, text to image, text to code or audio, etc. The first idea of the transformer was to take a large chunk of text, obtain an embedding, and then use a specific neural network for tasks like text classification or translation. But then, you can build sequence-to-sequence architectures that allow a neural network to receive a chunk of text, embed it into a real-value vector, and generate a completely different chunk of text from it.

For example, you can encode a sentence in English with a transformer that embeds it into a real-value vector and then decode it with another transformer that “speaks” French. The real-value vector in the middle represents the meaning of the text independent of language. So, you can have different encoders and decoders for various languages and translate any language pair.

A remarkable phenomenon is that you can train embedings in pairs of languages like English-Spanish and German-French and then translate from English to French without ever training on translations from English to French. This is due to using a shared internal representation for all languages. The sequence-to-sequence transformer is a fundamental piece behind technologies like ChatGPT. The next step is training it on massive amounts of text and then fine-tuning it for specific tasks.

Large Language Models

Large language models are the latest development in statistical language modelling, evolving from N-Gram models, embeddings, and transformers. Thanks to innovations that efficiently accommodate thousands of words in memory, these advanced architectures can compute contextual embeddings for extensive text contexts. This capacity has increased continuously, with the first version of ChatGPT holding something like 4000 words, while recent models hold anything from 30 thousand to a couple million words in the context!

A significant change is the scale of data on which these models are trained. BERT was trained on a vast dataset for its time, but it pales in comparison to GPT-2, 3, and 4. Large language models learn from a massive amount of internet text, including technical texts, books, Wikipedia articles, blog posts, social media, news, and more. This exposure to diverse text styles and content allows them to understand various mainstream languages.

Large language models, like GPT-2, generate text by predicting the next word in a sentence or paragraph, just like all previous language models. But when you combine the massive scale of the data and computational resources put into making these beasts of language models, and some clever tricks, they become something completely beyond what anyone thought possible.

GPT-2 was a huge leap forward in terms of coherent text generation. Given an initial prompt—say, the introductory paragraph of a fictional story—the model would generate token after token creating a mostly coherent story full with fictional characters and a plot. After a while it would start to diverge, of course, but for short fragments of text, this was already mindblowing.

However, things really exploded with GPT-3. At this size, emerging capabilities like "in-context learning" appeared, and this is where our story really begins.

How do LLMs work?

A generative language model, at its core, is just a statistical machine learning model trained to predict the continuation of a text sequence. Essentially, it's a prediction machine. You input a text prefix, run it through the model, and receive the most likely next token--a token is more or less a word or component of a word.

Actually, you don't really get just the next most likely token. The model provides a distribution across all possible tokens, giving you the probability of each one being the next continuation.

To use an LLM, we start with user input, like a query or text prefix, and run the model to get the next input token. We append it to the sequence and repeat the whole process until reaching a maximum number of tokens or the model predicts a special STOP token.

There are choices to make in this process. Choosing only the most likely continuation can quickly lead to repetitive predictions. Instead, you can choose from the top 50 most likely tokens at random, weighted by their probability. This injects some variety in the generated and is the reason why, for the same prompt, you can get different albeit semantically similar responses.

There are a few key parameters in this sampling process: the top K tokens to choose, the cumulative probability, and the temperature, which is the most relevant. The temperature is a parameter that affects the weights of the tokens you will pick for continuation. If the temperature is 0, you'll usually choose the most likely token. If it's higher, probabilities are smoothed out, making it more likely to choose less probable tokens. This increases the model's variability.

That's why some call high-temperature "creative mode" and low-temperature "precise mode." It has nothing to do with actual precision or creativity, just how deterministic the response to a given prompt will be.

From this perspective, you can already see why some people say language models are "just autocomplete on steroids". Indeed, that is the gist of their mechanics: you're completing a text sequence by adding one token at a time until you decide to stop. However, this is just scratching the surface. There is so much more involved in getting these models to behave in a useful way, and we will talk about some of those aspects in the next section.

But before moving on, here is a key insight from this explanation of how LLMs work: A language model always performs a fixed amount of computation per token.

This means that whatever limited form of "reasoning" can happen in an LLM, the depth and complexity of that reasoning is directly proportional to the number of total tokens the model processes. This implies two things:

If the input prompt is larger, the model will perform more computation before starting to compute its answer. This is part of the reason why more detailed prompts are better. But crucially, if the output is larger, the model is also doing more computation.

This is why techniques like chain-of-thought—and basically anything that makes a model "talk more"—tend to improve their performance at some tasks.

They have more compute available to do whatever reasoning they can do. If you ask a model a quick question and instruct them to give a one-word answer, the amount of compute spent producing that answer is proportional to just the input size. But if you ask the model to produce a step-by-step reasoning of the answer before the final answer, there is a higher chance you'll get a better answer just by virtue of spending more computation.

At the risk of anthropomorphizing too much, I like to summarize this insight as follows: LLMs only think out loud. If you want them to think better, get them to talk more.

So, this is how a language model works from a user perspective. Let's see how you build one.

How to Train your Chatbot

How do you make your language model work? There are three main steps.

Pre-training

The first step is called self-supervised pretraining. In this step, you take a raw transformer architecture with uninitialized weights and train it on a massive amount of data to predict the next token. You use a large corpus of data, such as news, internet blog posts, articles, and books, and train the model on trillions of words.

The simplest training method is next token prediction. You show the model a random text and ask it what the next token is. Take a random substring from the dataset, remove the last token, show the prefix to the model, and ask for likely continuations. Compute a loss function to determine how mistaken the model was in its predictions and adjust it slightly to improve future predictions.

So far, this is a standard machine learning approach. We call it self-supervised learning because the targets are not given by humans, but chosen automatically from the input. But deep down, this is just supervised learning at scale.

Now, that being said, scaling this training process to billions of parameters and trillions of tokens presents a massive engineering challenge. No single supercomputer in the world can handle training GPT-4 from scratch, so you must resort to distributed systems to split the model across used across hundreds or thousands of GPUs for extended periods of time, synchronizing different parts of the model across multiple computers is crucial for efficient training. This just means, while the conceptual part of training an LLM is pretty straightforward, it is nothing short of an engineering prowess to get build like GPT-4.

Once pre-training is completed, you have what is called a "base model", a language model that can continue any sentence in a way that closely resembles existing text. This model is already extremely powerful. Give it any prefix of text with any content whatsoever and the model will complete it with a mostly coherent continuation. It's really autocompletion on steroids!

However, these base models, as powerful as they are, are still very hard to prompt. Crucially, they do not understand precise instructions, mostly because their training data doesn't contain a lot of examples of instructions. They are just stochastic parrots, in a sense. The next step is to get tame them.

Instruction tuning

At this point, the LLM already has all the knowledge in the world somewhere hidden in its weights--metaphorically speaking--but it is very hard to locate any concrete piece of knowledge. You must juggle with transforming questions into the right prompts to find a pattern that matches what the model has seen.

The way to solve this problem is to include another training phase, but this time much shorter and focused on a very well-curated dataset of instructions and responses. Here, the quality is crucial, much more than the quantity. You won't teach the model anything new, you will just tune it to expect instruction-like inputs and produce answer-like outputs.

Once finished, you have what's called an instruction-tuned model. These models are much more robust and easy to prompt compared to the base model, and this is the point where most open-source models end. But this is not the end of the story.

Instruction-tuned models are still not suitable for publicly-facing products for one crucial reason: they can be coerced into answering anything at all, including producing biased, discriminatory, or hate speech and instructions on how to build bombs and deadly poisons.

Given base models are trained on the whole Internet, they are full of all the good and bad you can read online--although some effort is put into cleaning the pretraining dataset, but it's never enough. We must teach the model that some questions are better left unanswered.

Preference tuning

The final step is to fine-tune the model to produce answers that are more closely aligned with user preferences. This can and is primarily used to avoid biased or hate speech, and to reject any questions that are deemed unethical by the developers training the model. However, it also has the effect of making the model more polite in general, if you wish so.

The way this process works is to turn the problem from supervised learning into the real of reinforcement learning. In short, the main difference is that, while in supervised learning we give the model the correct answers (as in instruction tuning), in reinforcement learning we don't have access to ground truth answers.

Instead, we use an evaluator that ranks different answers provided by the LLM, and a feedback loop that teaches the LLM to approximate that ranking. In its original inception, this process was performed with a human evaluator, thus giving raise to the term "reinforcement learning with human feedback", but since including humans makes this process slower and more more expensive, smaller organizations have turned to using other models as evaluators.

For example, if you have one strong model, like GPT-4, you can use it to rank responses by a smaller, still in-training model. This is one example of a more general concept in machine learning called "knowledge distillation" in which you attemp to compact the knowledge of a larger model into a smaller model, gaining in efficiency without sacrificing too much in performance.

And finally, we have now something that works like GPT-4. The process was long and expensive: a massive pretraining following by a carefully curated instruction tuning and a human-backed preference tuning. This is the reason why so few organizations have the resources to train a state-of-the-art large language model.

What can LLMs do?

Now that we understand how language models are built, let's turn our attention to their capabilities. As we've seen so far, base models are, ultimately, just autocompletion models. Given an initial prefix, they can produce a mostly coherent continuation that is plausible as far as the data and the training procedure allow.

But autocompletion is far from the only task you can do with LLMs. As we will see in this chapter, a sufficiently powerful autocompletion engine can be coerced into performing many disparate tasks. Combine this with task-specific fine-tuning, and you can turn a chatty, hallucination-prone LLM into a powerful tool for many domains.

We will start by examining what base models can do since, ultimately, all fine-tuning can do is unlock existing capabilities, making them easier to prompt. Then, we will survey many specific tasks for which LLMs can and have been used.

What can base models do?

As cool as it sounds, autocompletion on steroids doesn't ring like anything smart, right? Well, it turns out that if you are very, very good at completing any text prefix, that implies you must be good at a wide range of cognitive tasks.

For example, suppose you want to build a question-answering engine. Take a question like "Who is the current president of the United States" and turn it into a prompt like "the current president of the United States is...". If you feed this to a powerful base LLM, the most likely continuation represents the correct answer to the question. This means autocomplete on steroids gives you question answering for free.

And you can do this for a whole lot of tasks. Just turn them into an appropriate prefix and continuation. Do you want to translate a sentence? Use the prompt like "An English translation of the previous sentence is..." Do you want to summarize a text? Use a prompt like "A summary of the previous text is..." You get the point.

But it goes much further than that! The scientists at OpenAI discovered that models the size of GPT-3 and above were capable of inferring the semantics of a task given examples without explicitly telling them what the task was. This is called in-context learning, and it works wonders. For example, if you want to use an LLM for sentiment analysis, you can use a prompt like the following.

Comment: This movie was so good!
Sentiment: Positive

Comment: This movie really sucks.
Sentiment: Negative

Comment: The book was better.
Sentiment: Neutral

Comment: I couldn't stop looking at the screen!
Sentiment:

That is, you build a prompt with a few examples of inputs and outputs and feed that to the LLM, leaving the last input unanswered. The most likely continuation is the right answer to the last input, so provided the base model has seen similar tasks in its training data, it will pick up the pattern and answer correctly most of the time.

In-context learning is a surprising discovery at first, but when you look deep down, it makes total sense. Since base LLMs are completion machines, provided they have seen examples of some arbitrary task in their training set, all you need to do is come up with a text prefix that makes the model "remember" that task. And that prefix is often just a set of examples of a given task because that is actually what is stored in the LLM weights: a loosely and implicitly connected set of similar text fragments.

In a sense, the input to the LLM is a key to retrieving a part of its training set, but not in an accurate way. Since LLMs only store correlations between words, anything you "retrieve" from an LLM is a fuzzy approximation and aggregation of several (possibly millions) of similar training examples. For this reason, we say base models already "know" everything, but it's very hard for them to "remember" it, because you have to find the right key--i.e., the right context prefix.

But what if we could teach the LLM that some arbitary instruction is equivalent to the right key for a given task? That is exactly what instruction tuning is about. By showing the LLM input/output pairs of, this time, precise instructions and the corresponding answer, we are rewiring some of its parameters to strengthen the correlation between the instruction and the response.

In a sense, fine-tuning is like finding a path between the input space and the output space in the base model's fuzzy web of word correlations and connect those two subspaces of words with a shortcut, so next time you input the instruction, the LLM will "remember" where is the appropriate answer.

If this sounds overly anthropomorphic, it is because we have stretched the analogies a bit to make it easier to understand. In reality, there is no "remembering" or "knowing" happening inside a large language model, at least not in any way akin to how human memory and reasoning work. I have written extensively about this difference and its implications and will continue to do so in future posts.

For the time being, please be cognizant that any analogy between LLMs and human brains is bound to break pretty soon and cause major misunderstandings if taken too seriously.

Use cases for fine-tuned LLMs

With proper fine-tuning in a concrete domain, you can turn LLMs into task-specific models for a huge variety of linguistic problems. In this section, we'll review some of the most common tasks for which LLMs can be deployed.

When discussing the use cases of fine-tuned LLMs, we don't talk about an "input prefix" anymore because even if, technically, that is still what we are feeding the LLM, the response is not necessarily a direct, human-like continuation of the text. Instead, depending on which dataset it was fine-tuned, the LLM will respond with something that looks more like an answer to a question or an instruction than a pure continuation. Actually, if you give a fine-tuned LLM like GPT-4 an incomplete text prefix, it will often reply back with something like "I didn't understand you entirely, but it appears what you are trying to do is [...]" instead of casually continuing where you left.

Thus, it is often best to interpret this process as "prompting" the LLM with an instruction, and this is the reason why the input text is called a "prompt", and the process of designing, testing, and optimizing these prompts is called, sometimes undeservedly, "prompt engineering".

Text generation

The simplest, most straightforward use case for large language models is of course text generation, whether for fictional content as for technical articles, office work, homework, emails, and anything in-between. But instead of using a base model, where you have to provide a prefix to continue, an instruction-tuned model can be instructed directly to write a paragraph, passage, or even a short essay on a given topic. Depending on how powerful and well-trained the model is, you can even provide hints about the intended audience, the complexity of the language to use, etc.

Text generation--and all instructions in general--often works better the more descriptive the prompt. If you simply ask the LLM to "tell me a fairy story", yes, it will come up with something plausible, and it might even surprise in the good way. But you most likely want to have finer control over the result, and thus crafting a well-structured and informative prompt is crucial. In @sec-prompting we will learn the most basic strategies to create effective prompts.

A common issue in text generation, especially in longer formats, is that the LLM can and will often steer away from the main points in the discourse. The longer the response, the most likely some hallucinations will happen, which may be in the form of incoherent or plain contradictory items, e.g., characters acting "out of character" if you're generating fiction.

A battle-tested solution for generating coherent, long-form text is the divide-and-conquer approach. Instead of asking for a full text from the begining, prompt the LLM to first generate an outline of the text, and then, sequentially, ask it to fill in the sections and subsections, potentially feeding it with previously generated content to help it mantain consistency.

Summarization

Summarization is one of the most common and well-understood use cases of LLMs. In a sense, it is a special case of text generation--what isn't, right?--but it has specific quirks that merit a separate discussion. In general, LLMs excel at summarizing. After all, that's what they've been implicitely trained to do: construct a statistical model of the whole internet, which is rather, in a sense, a summary of the whole human knowledge.

However, summarization isn't a trivial problem. Besides the usual concerns about the audience, complexity of the language, style, etc., you will probably also want to control which aspects of the original text the LLM focuses on. For example, rather than a simple compactation of the text, you might want a summary that emphasizes the consequences of whatever is described in the original text, or that highlights and contrasts the benefits and limitations. This is a more abstract form of summary that produces novel value, beyond just being a shorter text.

There are important caveats with summarization, though. LLMs are very prone to hallucination, and the more you push the boundary between a plain summary and something closer to critical analysis, the more the LLM will tend to ignore the original text and rely on its own pre-trained knowledge.

And just like before, the best way to counteract any form of rebellious generation is to be very intentional in your prompt and make it as structured as necessary. For example, you can first ask the LLM to extract the key points, advantages, and limitations. Then, ask it to cluster the advantages and limitations according to your criteria. Only then can we ask it to provide a natural language summary of that semi-structured analysis. This gives you finer control over the end result and will tend to reduce hallucinations while being easier to debug since you can see the intermediate steps.

Translation & style transfer

The text-to-text transformer architecture (the precursor and core component of the modern language model) was originally designed for translation. By encoding the input sentence into a latent space of word correlations detached from a specific language and then decoding that sentence in a different vocabulary, these models achieved state-of-the-art translation in the early 2018s. The more general notion of style transfer is, deep down, a translation problem, but instead of between English and French, say, between technical and plain language.

Modern LLMs carry this capability, and will be more than enough for many practical translation tasks. However, beware that plenty of studies show that LLM translation are often poorer in many linguistic notions from professional translations. Translation is an art, as much or more than it is a science. It involves a deep knowledge of the cultural similarities and differences between readers of both languages, to correctly capture all the nuances that even a seemingly simple phrase can encode.

That being said, LLMs can help bridge the gap for non-native speakers in many domains where you don't need--or can't hope for--a professional translation. An example is inter-institutional communication, e.g., emails from co-workers who don't speak your native language. In these cases, you must also be careful nothing important is lost in translation, literally, but as long as everyone is aware of the limitations, this is one of the most practical use cases for LLMs.

Structured generation

Continuing with the topic of text generation capabilities, our next stop is generation from structured data. This is one specific area where LLMs come to mostly solve a long-standing problem in computer science: to generate human-sounding explanations of dry, structured data.

Examples of this task are everywhere. You can generate a summary of your calendar for the day, and pass it to a speech synthesis engine, so your personal assistant can send you every morning an audio message reminding you what you have to do, with cute linguistic cues like "Oh, and on the way to the office, remember to pick up the your wife's present." We will see an example of this functionality in @sec-planner.

Other examples include generating summaries of recent purchases for a banking app or product descriptions for an online store—basically anywhere you'd have a dashboard full of numbers and stats, you can have an LLM generate a natural language description of what's going on. You can pair this capability with the super skills LLMs have for question answering (at least when the answer is explicit in the context) to construct linguistic interfaces to any of number of online services or apps.

Text classification

Text classification is the problem of categorizing a text fragment—be it a single sentence, a whole book, or anything in between—into one of a fixed set of categories. Examples vary from categorizing comments as positive/neutral/negative, determining if an email is spam or not, or detecting the tone and style of a text, to more specific tasks like extracting the intended intention from a user, e.g., chatting with an airline bot.

To have an LLM correctly and robustly classify your text, it is often not enough to just instruct it and provide the intended categories. The LLM might come up with a new category you didn't mention just because it makes sense in that context. And negative instructions, in general, don't work pretty well. In fact, LLMs are lousy at interpreting negative instructions precisely because of the underlying statistical model. We will see in @sec-reasoning why this is the case.

Instead of a dry, zero-shot instruction, you can improve the LLM classification capabilities substantially with a few examples (also called a k-shot instruction). It works even better if you select the examples dynamically based on the input text, a procedure that eerily similar to k-NN classification but in the world of LLMs. Furthermore, many LLMs tend to be chatty by design, and will often fail to provide a single word classification even if you instruct it to. You can mitigate this by using a structured response prompt, as seen in @sec-prompting.

Structured data extraction

A generalization of text classification is the problem of structured data extraction from natural language. A common example is extracting mentions of people, dates, and tasks in a text, for example, a transcription from a video meeting. In the more general case, you can extract any entity-relation schema from natural text and build a structured representation of any domain.

But this capability goes much further. If you have any kind of structured input format--e.g., an API call for any online service--you can instruct (and probably k-shot) an LLM to produce the exact JSON-formatted input given a user query. This is often encapsulated in modern LLM providers i a functionality called "function calling", whic which we will explore in @sec-function-calling.

As usual, the main caveat with structured generation is the potential for subtle hallucinations. In this case they can be in two forms. The simplest one, when the LLM fails to produce the expected format by, e.g., missing a key in JSON object or providing an invalid type. This type of error is what we call a syntactic hallucination and, although anoying, is often trivial to detect and correct, even if just by retrying the prompt.

The second form of hallucination is much more insidious: the response can be in the right format, and all values have the right type, but they don't match what's in the text. The LLM hallucinated some values. The reason this is a huge problem is because detecting this form of semantic hallucination is as hard to solve as hallucinations in general. As we'll see in @sec-hallucinations, we simply have no idea how to ensure an LLM always produce truthful responses, and it might be impossible even in principle.

Question answering

Question answering is one of the most surprising capabilities of sufficiently large language models. To some extent, question answering can be seen as a form of retrieval, where you ask about some facts explicitly mentioned in the training set. For example, if you ask, "Who wrote The Illiad" it is not surprising, given what we know of LLMs, that a fine-tuned model can easily generate "Homer" as the most plausible response. The sentence "Homer wrote The Illiad" must have appeared thousands of times in different ways in the training set.

But modern LLMs can go way beyond simply retrieving the right answer to a trivia question. You can ask questions that involve a small set of reasoning steps, combining facts here and there to produce a response that is not, at least explicitly, in the training set. This is rather surprising because there is no explicit reasoning mechanism implemented in LLMs. All forms of reasoning that can be said to happen are an emergent consequence of learning to predict the next token, and that is at least very intriguing.

In any case, as I’ve argued many times, the statistical modelling paradigm has some inherent limitations that restrict the types of reasoning that LLMs can do, even in principle. This doesn't mean that, in practice, it can't work for the types of problems you encounter. But in its most general form, long-term reasoning and planning are still an open problem in artificial intelligence. I don't think LLMs alone are equipped to solve it.

You can, however, plug LLMs with external tools to enhance its reasoning skills. One of the most fruitful research lines is to have them generate code to solve a problem, and then run it, effectively making LLMs Turing-complete, at least in principle, even if in practice they may fail to generate the right code. Which leads us to the next use case.

Code de generation

Since LLMs are trained to autocomplete text, it may not be that surprising that, when feed with enough training examples of code in several programming languages, they can generate small snippets of mostly correct code. However, for anyone who codes, it is evident that writing correct code is not as simple as just concatenating plausible continuations. Programming languages have much stricter syntax rules that require, e.g., to close all parenthesis and to use explicit and very tight naming conventions. Failing to produce even a single semicolon in the right place can render a program unusable.

For this reason, it is at least a bit surprising that LLMs can code. More surprising still is that they can not only autocomplete existing code but generate code from scratch given natural language instructions. This is one of the most powerful capabilities in terms of integrating LLMs with external tools because code is, by definition, the most general type of external tool. There is nothing you can do on a computer that you can't do with code.

The simplest use case in this domain is, of course, using LLMs as coding assistants embedded in developer tools like code editors. But this is just scratching the surface. You can have an LLM generate code to solve a problem it would otherwise fail to answer correctly--e.g., perform some complex physics computations. Code generation allows an LLM to analyze large collections of data by computing statistics and running formulas. You can even have an LLM generate the code to output some chart, and voilá, you just taught the LLM to draw!

Code explanation

Code explanation is the inverse problem of code generation: given some existing code, produce a natural language explanation or, more generally, answer questions about it. In principle, this is a form of question-answering that involves all the caveats about complex reasoning we have already discussed. But it gets harder.

The problem is the majority of the most interesting questions about code cannot be answered in general: they are undecidable, meaning no algorithm can exist that will always produce the correct response. The most poignant example is the question, "Does this function ever returns?". This is the well-known Halting problem, the most famous problem in computability theory, and the grandfather of all undecidability results. Similar questions, such as whether a variable is ever assigned or a method is ever called, are also undecidable in the general case.

And this is not just a theoretical issue. The Halting problem highlights one crucial aspect of computation: in the general case, you cannot predict what an algorithm will do without running it. However, in practice, as anyone who codes knows, you can predict what lots of your code will do, if only because it is similar to code you've written before. And this is where LLMs shine: learning to extrapolate from patterns to novel specific instances, even if the general problem is unsolvable.

To top it all, we can easily imagine an LLM that, when prompted with a question that seemingly cannot be answered from the code alone, could decide to run the code with some specific parameters and observe its results, drawing conclusions not from the syntax alone but from the execution logs. A debugging agent, if you will.

Final Remarks

These are the most essential high-level tasks where LLMs can be deployed, but they span hundreds, if not thousands, of potential applications. Text classification, for example, covers a wide range of applications, just changing the classification target.

One conclusion you can draw from this chapter is that LLMs are some of the most versatile digital technologies we've ever invented. While we don't know if artificial general intelligence is anywhere near, we're definitely one step closer to general-purpose AI—models that can be easily adapted to any new domain without research or costly training procedures.

However, language modelling is not magic. The above discussion has already given us a glimpse of some of this paradigm's fundamental limitations. In future posts, we will explore how these models learn compared to humans and what this difference entails regarding their limitations.

If you want to know more about language modelling in general, and LLMs in particular, feel free to check Mostly Harmless AI. It’s jam-packed with information (most of which is published in this blog already) on the good and the ugly parts of LLMs, and lots of advice on how to get the best out of them.

Welcome to another issue of Mostly Harmless AI, the section of where I share educational articles for the general audience on the potential and limitations of Artificial Intelligence. This article is a bit of a mashup of several past articles, with new insights and much better organization. It is my attempt to put a final dot in the discussion of whether LLMs can or cannot reason—at least for a while. I hope you enjoy it.

This article first appeared in , a newsletter hosted by that offers deep dives into recent AI topics, and much more. Please check the original article for additional insights and opinions from Michael.

See the original article

This version has only minor syntax and grammar fixes.

— Alejandro

Reasoning models are the latest trend in AI.

It all began with OpenAI o1, a model designed to "think out loud" before arriving at answers, mimicking human thought processes in an almost uncanny way. Then came DeepSeek R1, which has garnered massive attention for its impressive performance in reasoning tasks, which sent shock waves across the whole AI industry and sparked a few conspiracy theories.

And now we're seeing the upgrade from o1 to o3 in real-time (yes, these people are really bad at naming things), another reasoning model from China, and Google and Meta throwing their hats into the fight. Everyone runs to get their "reasoning" model out as quickly as possible.

For many, these models seem to be more than just iterations of the previous paradigm; they represent a qualitative leap, particularly in tackling challenges that require logical reasoning. Across many complex benchmarks, reasoning-enhanced models have shown impressive accuracy at the cost of increased response time, which seems like a fair deal.

The big shift is trading more computation during inference—making the model "think out loud" for, sometimes, several minutes before deciding on an answer—for a decreased rate of hallucinations and reasoning mistakes and an improved capacity for producing long chains of mostly sound logical arguments to arrive at a non-trivial (read, pattern-matched) answer.

However, despite their impressive performance, these models are not infallible. Numerous examples illustrate their limitations, where they deviate from the correct thinking path, arrive at incorrect conclusions, and even occasionally contradict themselves. In particular, these models seem to struggle with formal reasoning: mathematical and logical problems that require not as much real-world or common-sense knowledge but rather the precise application of logical inference rules.

This raises some crucial questions: Is the path to flawless reasoning merely a matter of superior data and extended training? Can we bridge the gap to achieve the kind of formal reasoning from natural language we've seen in sci-fi AIs—like Data from Star Trek or the hyperrational robots in Asimov's narratives? Or is there something fundamentally limiting within the architecture of large language models that restricts their capability for comprehensive logical reasoning in natural language?

In this article, I will argue that LLMs have intrinsic limitations that hinder their reasoning abilities. The core of the argument boils down to the combination of stochastic language modelling with a fixed computational architecture, which renders LLMs incapable of provably correct general-purpose formal reasoning.

I will attempt to explain this in clear and intuitive terms and address the most compelling counterarguments to this perspective. This article summarizes and expands on my views in several previous articles. It's part of my ongoing attempt to understand where we are heading with LLMs regarding reasoning.

While much of this discussion in this article applies primarily to first-generation LLMs—anything before OpenAI o1—I will also explore how these fundamental constraints extend to newer reasoning-enhanced models. If my arguments hold, nothing short of a novel paradigm can lead to AGI. In fact, I will argue that perfect reasoning from natural language may be computationally unsolvable, even in principle.

But before moving on, let me clarify something. I understand this topic is very controversial, and I know I will receive a lot of criticism, especially from some of the most ardent believers in the immediacy of AGI.

To be clear, I'm not claiming AGI is impossible. On the contrary, I'm firmly on the computationalist side here, and I believe machines can achieve the same and perhaps even superior levels of intelligence as any living, organic being. And I'm confident LLMs will play a significant role in the AGI breakthrough. That's why I am among the many researchers working on better ways to make LLMs interact with other computational systems to enhance their capabilities. I’m just sceptical that all that’s left is throwing more GPUs and using the same techniques we already have.

With that out of the way, let's explore what it means to say that LLMs are fundamentally limited in formal reasoning. This will be a long read, so buckle up!

What Do We Mean by Reasoning in LLMs

"Reasoning" is a very loaded term, that means a lot of things for different people—even for different groups of people who should agree on these definitions. So, I will attempt to define precisely what I mean by the phrase "LLMs cannot reason.”

In the Artificial Intelligence field, when we claim LLMs (or any computational system) can or cannot "reason", we are, for the most part, not talking about any abstract, philosophical sense of the word “reason”, nor any of the many psychological and sociological nuances it may entail. There are many ways to define reasoning, from common sense to analogical to emotional to formal. In this article, I will focus on a precise, quantifiable, simplified notion of reasoning that comes straight from math.

Reasoning is, simply put, the capacity to draw logically sound conclusions from a given premise. Formal reasoning is when you follow a formalized method with hard rules to arrive at a conclusion. The most important tool for formal reasoning in math and science is logic, and it provides two fundamental reasoning modes: deduction and induction.

Induction is somewhat problematic for AI (and humans!) because it involves generalizing claims from specific instances, and thus, it requires some strong assumptions. In contrast, deduction is very straightforward. It is about applying a set of predefined and agreed-upon logical inference rules to obtain new provably true claims from existing true claims. It is the type of reasoning that Sherlock Holmes is most famous for, and the thing mathematicians do all day long when proving new theorems.

Thus, in the remainder of this article, when I say "LLMs cannot reason", I’m simply saying LLMs cannot perform logical deduction; that is, there are well-defined, computationally solvable deduction problems they inherently cannot solve. As I will attempt to convince you, this is not a value judgment or an informed opinion based on experience. It is a straightforward claim provable from the definition of deductive reasoning and the inherent characteristics of LLMs given by their architecture and functionality.

Now, if this definition of “reasoning” sounds like an oversimplification, well… in a sense it is. I'm not addressing other forms of non-formal reasoning that are also paramount in human thinking, like abduction or common-sense and analogical reasoning. But formal reasoning, and particularly deductive reasoning, is the basis of mathematical and scientific research and crucial in informed decision-making, so any attempt to build an AI system useful for these tasks must be able to perform this mode of reasoning.

Before moving on to why LLMs cannot reason, let me address the most common counterargument I encounter whenever this topic is raised in non-academic contexts.

Can Humans Truly Reason?

Here is an argument you will hear a lot, and maybe even have made yourself:

“Sure, LLMs cannot really reason, but neither can humans, right? So what's the big deal.”

I mean, humans can be stupendously irrational. We are prone to so many biases that, while useful in the biological and sociological context in which we evolved, in the modern world, are often more than not an obstacle to rational, logical thinking.

Despite this, the argument that since humans are not perfectly rational, it is OK for LLMs to also not be is flawed on many levels, so let’s unpack it.

First, while humans can make errors in reasoning, the human brain definitely possesses the capacity for open-ended reasoning, as evidenced by the more than 2000 years of solid math we have collectively built. Moreover, all college students—at least in quantitative fields—at some point have to solve structured problem-solving exercises that require them to apply logical deduction to arrive at correct conclusions, such as proving theorems.

So, while humans can be pretty stupid at times, we are certainly capable of the most rigorous reasoning when trained to do so.

But even more importantly, this assertion is a common case of whataboutism. Why does the fact humans can’t do something immediately make it OK for a piece of technology to fail at it? Imagine we did this with all our other tech. Sure, that aeroplane fell down and killed 300 people, but humans can’t fly. Or yes, that submarine imploded, but humans can’t breathe underwater. Or that nuclear power plant melted, but humans can’t stand 3000 degrees of heat, so what’s the big deal?

Obviously, we don’t do that. We compare any new piece of technology with our current best solution, and only if the new thing improves upon the old—at least on some metrics—do we consider the investment worthwhile. We replaced horses with cars because cars improved the previous best solution to individual transportation, at least on some fundamental metrics, if not all.

Granted, we often compare AI capabilities to human capabilities, but this is only because humans are the gold standard for the types of problems we often want AI systems to solve. So, we compare LLMs' capacity to generate creative stories, engage in open-ended dialogue, or provide emphatic customer assistance with humans because humans are the best at these tasks.

However, there are well-established systems—such as traditional SAT solvers—that excel in structured logical deduction and reasoning tasks. These systems are designed with rigorous validation mechanisms that ensure correctness and reliability in their outputs. They are basically flawless and incredibly fast.

So, in terms of the capability to perform purely logical, perfectly valid deduction, we already have a computational solution that sets the bar. Nothing short of provably correct deduction is good enough. The problem is these formal reasoning systems don't understand natural language, of course, and that's why we want LLMs to bridge the gap.

Why LLMs Are Incapable of Formal Reasoning

Let's move on to the main limitations of current large language models that prevent general-purpose, provably correct deductive reasoning. I'm mostly thinking about LLMs implemented in the prevalent paradigm of the Transformer architecture, but these arguments apply to anything implemented using some sort of context-limited, probabilistic, sequential language model designed to run on GPUs.

The Argument from Stochastic Modelling

The first limitation of stochastic language models regarding reasoning is precisely their stochastic nature. These models generate outputs based on probabilistic predictions rather than deterministic logical rules. This means that even a well-structured prompt can and will yield different responses on different occasions due to the randomness of the sampling process.

Now, stochastic sampling is a necessary feature of LLMs, and not a bug. It is the fundamental reason that LLMs can produce varied, plausible, humanly-sounding text at all. It is also the only way we know how to capture in software, at least approximately, the many nuances and ambiguities of natural language.

That being said, stochastic sampling is a problem for deductive reasoning. For starters, an LLM might arrive at a wrong conclusion purely by chance, even if the right answer was the most likely continuation. This is, once again, the infamous problem of hallucinations.

To alleviate this, we may attempt to set the sampling temperature to zero, effectively forcing the model to fix the output for a given input. Thus, there will be no more different answers to the same prompt.

However, the underlying model is still probabilistic; we’re just greedily sampling the most likely continuation. The problem is that the mapping between input and output hinges on a probabilistic distribution that encodes correlations between elements in the input and corresponding elements in the output.

The reason this is problematic is simple: any probabilistic model of language that can generate new sentences that didn't exist in the training data—that is, that can generalize at all—has to be, by definition, a hallucination machine. Or, to use ’s much better terminology: a confabulation machine.

You can think about it like this: if you want a language model to generate novel sentences, then it must be able to produce sentences it doesn't really know, but somehow believes are correct.1

The way a stochastic language model "believes" a sentence is correct, is because the probability of seeing this exact sentence is high. Crucially, for novel sentences, this probability is high because there is some similarity to actual sentences in the training set.

This means any stochastic language model that's useful at all needs to approximate correctness through plausibility. Sentences close (in a precise mathematical sense we won't get into here) to what the LLM saw in the training set are thus considered correct. Sentences that are too dissimilar to anything in the training set are considered incorrect. But the frontier is fuzzy; there is no explicit threshold to distinguish what's true from what's not. All a stochastic language model can actually model is degrees of plausibility.

And there is the problem. Math and logic are not fuzzy2. There is no plausibility involved. A reasoning chain is either correct or incorrect. And more importantly, a tiny change in a single word—like adding a single "no" somewhere in a huge prompt—can completely shift the validity of a logical claim.

Whenever you approximate crisp mathematical correctness with fuzzy plausibility, you will have true and false claims close enough to each other such that a stochastic model cannot effectively distinguish them.

Now, this discussion has important nuances, so let's analyze some interesting arguments about the role of randomness in reasoning.

Is Randomness a Bug or a Feature?

A common criticism of this argument regarding the stochastic nature of language models is that, in fact, randomness is essential in problem-solving and a crucial feature of many of the same SAT solvers I pretend to compare LLMs with. How hypocritical it is to posit randomness as a limitation when the most effective deductive reasoning algorithms are essentially random search algorithms!

This is true, but only partially, and it makes all the difference. So, let me explain.

Randomness plays a vital role in many computational problem-solving techniques, particularly in search algorithms for hard (read NP-complete or NP-hard) problems. Modern SAT solvers, for example, often employ randomized search strategies to efficiently explore vast solution spaces. By introducing randomness into the search process, these solvers can escape local optima and discover satisfactory solutions more quickly than deterministic methods might allow.

However—and here comes the crucial difference—using randomness in the search process does not imply that the entire reasoning process is inherently unreliable. Randomness is confined to the search phase of problem-solving, where it helps identify potential solutions—potential reasoning paths. However, once a candidate solution is found, a deterministic validation phase kicks in that rigorously checks the correctness of the proposed reasoning path.

This distinction between the search and validation phases is paramount in understanding how randomness contributes to effective problem-solving in general. During the search phase, algorithms may employ random sampling or other stochastic methods to explore possibilities and generate potential solutions. This phase allows for flexibility and adaptability, enabling systems to navigate complex landscapes of potential answers.

However, once a potential solution has been identified, it must undergo a validation process grounded in deterministic logic. This validation phase involves applying predefined, deterministic rules to confirm that the proposed solution meets all necessary criteria for correctness. As a result, any solution that passes this validation step can be confidently accepted as valid, regardless of how it was generated.

Here is a silly metaphor to illustrate this problem. If you sit a million monkeys in a million typewriters, at some point, one of them will randomly produce Romeo and Juliet. However, you need Shakespeare to filter the garbage out and decide which of the million pamphlets to publish.

What this means is randomness is good for exploring hypotheses but not for deciding which one to accept. For that, you need a deterministic, provably correct method that doesn’t rely on probabilities---at least if you want to solve the problem exactly.

However, in stark contrast to traditional problem-solving systems like SAT solvers, LLMs lack a robust validation mechanism. While they can generate coherent and contextually relevant responses based on probabilistic reasoning, some of which may be correct reasoning chains, they do not possess a reliable method for verifying the accuracy of those outputs. The verification process is also stochastic and as subject to hallucinations as the generation process, rendering it effectively unreliable.

Therefore, since LLMs evaluate their own outputs using the same probabilistic reasoning they employ to generate them, there is an unavoidable, although perhaps small, risk that incorrect conclusions will be propagated as valid responses.

The monkeys are also the publishers.

The Argument from Bounded Computation

The second argument concerns the computational architecture of current language models. By design, LLMs spend a fixed amount of computation per token processed. Thus, the amount of computation an LLM does before it produces the first output token is a function of just two numbers: the input size and the model size.

So, if you ask an LLM to produce a yes or no question for a logical puzzle, all the “thinking” the model can do is some fixed—albeit huge—number of matrix multiplications that only depend on the input size.

Now, consider that you have two different logical puzzles with the same input size, i.e., the same number of tokens. But one is an easy puzzle that can be solved with a short chain of deduction steps, while the other requires a much higher number of steps. Here is the kicker: any LLM will spend exactly the same amount of computation on both problems. This can’t be right, can it?

A basic result in computational complexity theory is that some problems with very small inputs seem to require an exponentially high computational cost to be solved correctly. These are NP-complete problems, and most computer scientists believe there are no efficient algorithms to solve them. Crucially, a huge number of reasoning problems fall in this category, including the most basic logical puzzle of all—determining if a given logical formula can be satisfied.

When faced with an instance of an NP-complete problem, an LLM will produce an answer after a fixed amount of computation defined solely by the input size. Now, by sheer size, some larger models might just spend enough computation to cover many smaller instances of NP-complete problems.

As it happens, a huge constant function can be larger than an exponential function for smaller inputs. But crucially, we can always find instances of NP-complete problems that require, even in principle, a sufficiently large amount of computation to surpass the computational capacity of any LLM, no matter how big.

But this means something even more profound. Ultimately, LLMs are not Turing-complete systems but essentially very large finite automata. While they can handle a wide range of tasks and produce outputs that appear sophisticated, their underlying architecture limits the types of problems they can solve.

Turing completeness is the ability of a computational system to perform any computation given sufficient time and resources. Modern computers and many seemingly simple systems, such as cellular automata, are Turing complete systems. Ironically, LLMs are not.

The reason is simple. We know from computability theory that any Turing complete system must be able to loop indefinitely. There are some problems—some reasoning tasks—where the only possible solution is to compute, and compute, and compute until some condition holds, and the amount of computation required cannot be known in advance. You need potentially unbounded computation to be Turing complete.

And this is the final nail in the coffin. LLMs, by definition, are computationally bounded. No matter their size, there will always be problem instances—which we may not be able to identify beforehand—that require more computation than is available in the huge chain of matrix multiplications inside the LLM.

Thus, when LLMs seem to tackle complex reasoning problems, they often solve specific instances of those problems rather than demonstrating general problem-solving capabilities. This might just be enough for practical purposes—we may never need to tackle the larger instances—but, in principle, LLMs are incapable of truly open-ended computation, which means they are incapable of true reasoning.

But again, there is a lot of nuance in this limitation. For starters, this analysis applies very obviously to first-generation LLMs. But newer, so-called reasoning-enhanced LLMs work a little differently: they produce a potentially unbounded sequence of "thinking tokens" before deciding on the actual answer.

So let's examine this argument from bounded computational power in more detail.

Can LLMs Perform Unbounded Computation?

A common counterargument is that LLMs can be rendered Turing complete by integrating them with external tools, such as code generators or general-purpose inference engines. It is even easier to wrap them in a recursive loop that can simply call the LLM as many times as necessary.

And this is true. You can trivially make an LLM Turing-complete, in principle, by duct-taping it with something that is already Turing-complete. You can also build a flame thrower with a bamboo stick, some duct tape, and a fully working flame thrower.

However, simply making LLMs Turing complete in principle does not guarantee that they will produce correct or reliable outputs. Integrating external tools or clever self-recursion introduces yet another layer of complexity and potential points of failure.

We need to address two main strategies in this regard: prompting and function calling. Let's tackle them one at a time.

Prompt-based Techniques for Reasoning

Chain-of-thought prompting is the most basic way to increase the computation of LLMs. By guiding models to articulate intermediate reasoning steps before arriving at a final answer, CoT prompting helps decompose complex problems into manageable parts. This method has improved performance across various reasoning tasks, such as arithmetic and commonsense reasoning.

CoT makes the LLM "think harder" by forcing the model to produce what we can consider “internal thought” tokens. Thus, we may view it as a way to perform additional computation on the input before deciding on the response.

This is precisely what modern reasoning-enhanced models like o1 and R1 are doing: automating the chain of thought prompt by baking it into the training process. However, despite its advantages, CoT prompting remains insufficient for several reasons.

On the one hand, CoT doesn't address the fundamental limitation of hallucinations. The stochastic nature of LLMs means that even with CoT prompting, outputs can vary across different runs due to randomness in the generation process. This variability can lead to inconsistent reasoning outcomes, undermining the reliability of the model's responses.

On the other hand, CoT extends the computation budget by a finite amount. To achieve true unbounded computation, we need a cyclic scheme in which the LLM is prompted to continue thinking, potentially indefinitely, until satisfied.

A potential solution for this problem is the intuitive approach of self-critique, which involves evaluating and refining an LLM's responses with the same model, using prompts that instruct the model to read its previous output, highlight potential errors, and try to correct them. A form of after-the-fact chain-of-thought, if you might.

However, research also shows significant limitations in the effectiveness of this self-critique capability.

While LLMs can generate multiple ideas and attempt to critique their initial outputs, studies indicate that they cannot often meaningfully self-correct. Research also shows that self-correction techniques in LLMs are heavily contingent on the availability of external feedback. In many cases, LLMs perform better when they have access to an external verifier or additional context rather than relying solely on their internal reasoning capabilities.

And even more interestingly, attempts at self-critique can sometimes degrade performance rather than enhance it. Studies have shown that when LLMs engage in self-critique without external validation, they may generate false positives or incorrect conclusions. If you push harder, you can easily fall into a cycle of self-reinforcement of invalid or erroneous arguments, making the LLM increasingly more certain despite it getting worse and worse.

Ultimately, this circles back to the already discussed issue of relying on randomness for validation. If we attempt to get the model to tell us when it is certain it got the answer right, we are doomed. CoT and self-critique are great strategies for exploring different hypotheses and generating multiple possibilities, but they alone cannot reliably produce a provably correct conclusion all the time.3

Function Calling and External Tools

Integrating external tools, such as reasoning engines or code generation systems, into large language models represents a promising—and, for me, the only really viable—approach to enhancing their reasoning capabilities.

Connecting LLMs to external reasoning engines or logical inference tools makes it possible to augment their reasoning capabilities significantly. These tools can handle complex logical deductions, mathematical computations, or even domain-specific knowledge that the LLM might not possess inherently. Similarly, external code generation systems enable LLMs to produce executable code for specific tasks.

By leveraging these external resources, LLMs can potentially overcome some of their inherent limitations in logical reasoning and problem-solving. For starters, an external inference engine will be Turing-complete, so we scratch that problem down, right?

Not so fast. Unfortunately, this approach has many challenges, particularly regarding the LLM's ability to generate the correct input for function calls or code execution. It all circles back to the original sin of LLMs: stochastic language modelling leads inevitably to hallucinations.

First, the effectiveness of function calling or code generation hinges on the model's ability to interpret a task accurately and generate appropriate inputs. If the model misinterprets the requirements or generates incorrect inputs (e.g., if it hallucinates part of the inputs), the external tool will produce erroneous outputs or fail to execute altogether.

While external tools can, in principle, improve the reasoning capabilities of an LLM by providing structured logic and formal verification, they cannot compensate for LLMs' basic limitations in generating reliable output. Therefore, there is no formal guarantee that the final output from this integration will be logically sound or appropriate for the context, simply because of the age-old adage: garbage in, garbage out.

Subscribe now

How About Reasoning-Enhanced Models?

Finally, let's briefly tackle what this means for state-of-the-art reasoning-enhanced LLMs, such as OpenAI's o3 and DeepSeek R1. These models employ extended "thinking" phases during inference, generating multi-step reasoning chains, critiquing their own outputs, and iteratively refining conclusions. This approach reduces hallucinations and improves accuracy on benchmarks like mathematical problem-solving and logical puzzles, often outperforming earlier models by wide margins.

The mechanics of these enhancements are based on two key innovations. First is the chain-of-thought expansion, which allows models to explore branching reasoning paths for longer durations—sometimes minutes of computation—simulating deeper deliberation. Second is implementing self-critique loops: internal validation mechanisms where the model evaluates its intermediate conclusions against problem constraints.

This article by provides a much deeper and more technically detailed description of how these models work.

These techniques do improve practical performance by a large margin. It's much better to bake CoT into the training process than simply relying on human-crafted prompts. During training, we can teach the model to automatically produce mostly correct reasoning chains by using formally verifiable training tasks, such as coding and math problems.

However, during inference, the accuracy of these models remains fundamentally limited by the probabilistic architecture of the underlying LLM. The self-critique process itself relies on the same stochastic language modelling that generates initial answers; there is no deterministic verification step during training. The same reliability issues remain, perhaps significantly decreased but not completely eliminated.

These models represent an evolution rather than a revolution. They scale up existing chain-of-thought techniques through increased computation and mirror human-like error checking with self-critique mechanisms that, ultimately, lack mathematical guarantees of correctness. Performance gains come from better exploring the model's existing knowledge space rather than new reasoning capabilities.

So, despite the remarkable empirical improvements in reasoning-enhanced models, they are more a natural evolution of existing paradigms rather than a qualitative breakthrough. Their stochastic foundation ensures occasional failures will persist despite enhanced computing budgets. They are, indeed, very powerful tools for many real-world applications where absolute formal certainty is not required, but they cannot overcome their fundamental limitations when tasked with formal deductive reasoning tasks requiring strict accuracy guarantees.

Conclusions

The purpose of this article is to convince you of two claims:

1. Large Language Models currently lack the capability to perform a well-defined form of reasoning that is critical for important decision-making processes, including producing novel, scientifically valuable output.

2. We have absolutely no idea how to solve this in the near future within the prevalent and, so far, the only scalable language modelling paradigm.

This matters because there is a growing trend to promote LLMs as general-purpose reasoning engines. As more users begin to rely on LLMs for important decisions, to perform deep research on their behalf, and in domains where they cannot validate the results, the implications of these limitations become increasingly significant. At some point, someone will trust an LLM with a life-and-death decision, potentially with catastrophic consequences.

More importantly, the primary challenges in making LLMs trustworthy for reasoning are immense. Despite ongoing research and experimentation, we have yet to discover solutions that effectively bridge the gap between LLM capabilities and the rigorous standards required for reliable reasoning. Currently, our best efforts in this area are duct tape—temporary fixes that do not address the underlying limitations of the stochastic language modelling paradigm.

Now, I want to stress that these limitations do not diminish the many other applications where LLMs excel as stochastic language generators. In creative writing, question answering, user assistance, translation, summarization, automatic documentation, and even coding, many of the limitations we have discussed here are actually features.

The thing is, this is what language models were designed for—to generate plausible, human-like, varied, not-necessarily-super-accurate language. The whole paradigm of stochastic language modelling is optimized for this task, and it excels at it. It is much better than anything else we’ve ever designed. But when we ask LLMs to step outside that range of tasks, they become brittle, unreliable, and, worse, opaquely so.

The emergence of models like OpenAI’s o1 and o3 models, DeepSeek's R1, Gemini 2.0, and the many more we'll definitely see in the short term, seem like a significant step forward. And to a large extent, they represent innovative and creative approaches, and open up new avenues of research and applications.

However, we haven't yet seen any fundamentally new paradigm in logical reasoning with LLMs. Deep down, this is “just” a way to explicitly incorporate chain of thought prompting in a fine-tuning phase and teach the model via reinforcement learning to select mostly coherent paths of deduction. It's clever scaling of what already works fairly good, but not perfectly.

Thus, while definitely an impressive technical and engineering feat, these reasoning-enhanced models, and any future models based on the same paradigm, will continue to share the same core limitations inherent to all LLMs, only mitigated using some clever tricks. Hallucinations will still hamper any attempt at definitively solving formal reasoning with LLMs. It may be a matter of degree, and for some domains, good enough might just be good enough.

Ultimately, provably correct reasoning from natural language might even be an uncomputable problem. After all, natural language semantics are anything but formal, so going automatically from informal problem descriptions to formal, provably correct solutions might be impossible, even in principle. I have some ideas about why this may be the case that I'll share in a future article.

If I'm right, then maybe all we can hope for is a very clever, very efficient, never-tired, sometimes incorrect, not fully trustworthy AI assistant—less like Data and more like Bender minus the sarcasm. But we will never be able to completely remove expert human judgment from any life-or-death decision. And that might just be the best outcome.

What do you think?

You already know that neural networks are everywhere, and there’s a reason for that, beyond fad. And no, it’s not simply because they are “inspired in the human brain”—we already debunked this partial myth in our previous article.

The true strength of neural networks lies primarily in their nature as mathematical constructs that are extremely flexible and powerful. This makes it relatively easy for them to adapt to nearly any domain. Additionally, they excel at leveraging vast amounts of data and computational power.

The ability of neural networks to model complex relationships and learn from vast amounts of data stems from several key mathematical properties. This article will explore these strengths, including the universal approximation theorem, the role of inductive biases in different architectures, and how layers within a network perform representation and manifold learning. We’ll also discuss the versatility of neural networks in transforming various learning objectives into optimized loss functions.

Then, in Part II, we’ll see how these networks scale impressively with increasing data and compute resources, further solidifying their position as a cornerstone of modern AI.

Subscribe now

Universal Approximators

The Universal Approximation Theorem is a cornerstone of neural networks and machine learning. It asserts that a feedforward neural network with a single hidden layer containing a finite number of neurons can approximate any continuous function on a compact subset of inputs to any desired degree of accuracy, provided that the activation function is non-constant, bounded, and continuous.

Whoa, that was a mouthful. In short, a big enough but still very simple neural network can, in theory, learn any pattern you want to with an arbitrary degree of precision.

The theorem's history dates back to the late 1980s when researchers began formalizing the mathematical foundations of neural networks. Early work by researchers like George Cybenko in 1989 demonstrated that single-layer networks could achieve this approximation capability. Since then, various versions and extensions of the theorem have been developed, solidifying its importance in understanding the theoretical underpinnings of neural networks.

What does this mean in practical terms? The Universal Approximation Theorem implies that neural networks are incredibly versatile tools capable of modelling a wide array of functions, from simple linear relationships to intricate non-linear mappings. This flexibility allows them to be effectively applied across numerous domains, such as image recognition, natural language processing, and more.

However, while the theorem guarantees the existence of such approximations, it does not provide a method for finding them efficiently in practice. Thus, while neural networks can theoretically learn any function, achieving that in real-world scenarios often requires careful design and training strategies.

Architectural Flexibility

While the Universal Approximation Theorem assures us that a sufficiently large, fully connected feedforward neural network with a single hidden layer can approximate any continuous function, this approach is often impractical for real-world applications. Instead, we can leverage specialized structures and different types of layers to create neural networks tailored for specific tasks. This architectural flexibility allows us to exploit the unique characteristics of the data and the problem domain, enhancing performance and efficiency.

One prominent example is convolutional layers, commonly used in Convolutional Neural Networks (CNNs). These layers are designed to process grid-like data, such as images. By applying convolutional filters, they can detect local patterns and features, such as edges or textures, while maintaining spatial hierarchies. This structure is particularly effective for image recognition tasks, where understanding spatial relationships is crucial.

Another example is recurrent layers, found in Recurrent Neural Networks (RNNs). These layers are specifically designed to handle sequential data, such as time series or natural language. By maintaining a hidden state that captures information from previous inputs, RNNs can effectively model temporal dependencies and context. This makes them well-suited for tasks like language modeling and speech recognition.

As a final example, remember transformers, which have revolutionized natural language processing. Unlike traditional RNNs, transformers rely on self-attention mechanisms to weigh the importance of different input elements relative to one another. This allows them to capture long-range dependencies and contextual relationships more effectively than previous architectures. Transformers have become the backbone of many state-of-the-art models in NLP, enabling tasks such as translation and text generation.

By employing these specialized layers, we can create neural networks that don't attempt to approximate arbitrary functions but rather exploit the inherent structure of the problems we are trying to solve.

Representation Learning

One of the most powerful aspects of neural networks is their ability to perform representation learning, which can be understood as a sequence of increasingly abstract feature extraction mechanisms. This process allows neural networks to automatically discover and learn relevant features from raw data without requiring manual feature engineering. Essentially, each layer in a neural network transforms the input data into higher-level representations, capturing more complex patterns as the information flows through the network.

Consider an image classification task. When we analyze what each layer of a convolutional neural network (CNN) is learning, we can observe a fascinating progression. The initial layers typically act as simple feature detectors, identifying basic elements such as edges and textures in various orientations. These early detectors are crucial for understanding the fundamental building blocks of an image.

As we move deeper into the network, these simple features begin to combine into more complex shapes and patterns. For instance, the next layers might learn to recognise geometric shapes like circles and squares by aggregating the edge information detected in the earlier layers. Further down the line, these shape detectors merge into even more sophisticated representations, such as figure-like detectors that can identify parts of objects or specific patterns.

By the time we reach layers 20 or more in a deep CNN, the network has developed a highly abstract understanding of the input data. At this stage, it can accurately detect complex objects like dogs, cars, or houses based on the intricate features it has learned to recognise throughout its architecture.

This hierarchical approach to feature extraction means that almost any neural network designed for classification tasks can be viewed as a sequence of increasingly abstract and complex feature extractors.

Manifold Learning

Manifold learning is another insightful way to interpret what neural networks are doing during the learning process. When tackling problems like image classification, we can think of it as a complex instance of a nearest neighbour problem. For example, all images of cats share certain similarities, just as images of dogs do. However, this similarity is not immediately apparent in the input domain—the pixel values—because images that represent similar concepts (like two different cats) can be quite distant from each other in terms of pixel-by-pixel distance.

To understand this better, we can posit that a high-dimensional space exists where these images are represented more meaningfully. In this space, points corresponding to similar images are close together, while those representing fundamentally different objects—like dogs, ships, or houses—are far apart. The challenge is that this "true" image space is tangled and twisted, making it difficult to identify these relationships directly.

Manifold learning refers to the ability of neural networks to find a set of transformations that project the original data from the input space (e.g., pixels in image classification) into this complex high-dimensional space where similar objects (e.g., images of cats) cluster together. If we could untangle this manifold, we could perform a simple nearest-neighbour comparison in a more meaningful context. Neural networks do that implicitly.

We can thus view deep neural networks as a series of projections into increasingly complicated manifolds. Each layer in the network transforms the input data, gradually mapping it closer to this ideal space where similar objects are grouped together. The final layer of the network, right before the softmax classification, thus contains a very twisted and tangled projection of the original input, to the point it would be unrecognizable by humans. Still, it happens to be the projection that best clusters together similar objects.

Backpropagation

Training neural networks effectively hinges on the backpropagation algorithm, a pivotal method since its introduction in the 1970s. Backpropagation allows for the fine-tuning of weights within a neural network by computing how to adjust all parameters based on the error from the previous iteration. This feedback mechanism is essential for optimizing the network's performance, as it systematically reduces the error rate by adjusting weights to improve predictions.

The power of backpropagation lies in its ability to compute gradients efficiently, regardless of the network's size or complexity. By applying the chain rule of calculus, backpropagation calculates how weight changes affect the overall error function. This means that even in deep networks with many layers, backpropagation can determine the necessary adjustments for each weight, enabling training to an arbitrary degree of precision (provided the network has enough capacity, i.e., is big enough).

In theory, this makes all neural networks trainable, but in practice, achieving effective training often requires careful management of various factors. For instance, practitioners must navigate challenges such as vanishing and exploding gradients, which can impede learning in deep networks. Additionally, hyperparameter tuning and regularization techniques are often necessary to ensure convergence and prevent overfitting. We’ll tackle these problems in Part II.

Flexible Learning Objectives

Neural networks are trained using backpropagation, which requires a well-defined loss function to measure the learning error. This loss function approximates how far off the network's predictions are from the target values. The loss function must be differentiable for gradient descent to work effectively. This allows us to compute gradients and optimize the network's weights.

However, many learning objectives are not inherently differentiable. A prime example is classification error, often referred to as 0/1 loss. This type of loss is binary: you either classify an instance correctly or incorrectly, providing no gradient information for optimization. Fortunately, we can create differentiable approximations of such non-differentiable loss functions.

For instance, the binary cross-entropy loss is a commonly used differentiable approximation for 0/1 loss in binary classification tasks. It captures the essence of correct and incorrect classifications while allowing for a continuous range of error values. This enables the model to learn more effectively by providing meaningful gradient information even when predictions are not perfect.

Similarly, other tasks have their own tailored loss functions that facilitate learning. For example, a common loss function in regression tasks is Mean Squared Error (MSE), which measures the average squared difference between the predicted and actual values.

For multi-class classification problems, neural networks often use Categorical Cross-Entropy Loss, an extension of binary cross-entropy. This loss measures the dissimilarity between the predicted probability distribution and the true distribution over multiple classes, making it particularly effective for problems with many output categories.

Hinge loss is frequently employed in tasks like binary classification, which focuses on maximizing the margin between classes. This loss is also commonly used in support vector machines and some neural networks to ensure better separation between categories.

Contrastive loss is often used in more specialized applications, such as face recognition or metric learning. This loss function helps models learn embeddings by minimizing the distances between similar pairs of data points while maximizing the distances between dissimilar ones.

Each of these loss functions is designed to suit specific learning objectives while maintaining differentiability, ensuring that gradient descent can be applied effectively to a wide range of dissimilar tasks.

Conclusions

Neural networks are incredibly powerful and flexible mathematical constructs. They can approximate any continuous function, adapt to specific tasks through specialized architectures like convolutional, recurrent, and transformer layers, and automatically extract increasingly abstract features from raw data.

Additionally, they untangle complex data relationships by projecting inputs into high-dimensional manifolds where similar items cluster together. Finally, their ability to transform diverse learning objectives into differentiable loss functions enables effective optimization via gradient descent.

And we have a very powerful and general training algorithm—backpropagation—to ensure we can effectively make use of all these mathematical properties.

But while these strengths explain their theoretical power, they don’t fully account for their practical success. In our follow-up article, we’ll explore how neural networks are perfectly suited to scale with the increasing availability of data and compute power, which is key to their dominance in modern AI.

Unless you've lived under a rock for the last couple of years, you already know that neural networks are the workhorse of AI. They’re everywhere—driving advancements from voice assistants to self-driving cars. But what makes them so unique?

Neural networks are not just a passing fad. They are incredibly powerful and flexible tools that can learn complex patterns from vast amounts of data. While they were initially inspired by how our brains work, today’s neural networks have evolved far beyond that simple analogy.

But before we further explore neural networks' power, we need to clarify one thing: artificial neural networks are nothing like the brain. This crucial distinction is often overlooked.

In this article, I will attempt to demystify the notion that artificial neural networks are anything like biological brains. And then, in a follow-up article, we’ll dive into the mathematical characteristics that give them flexibility and discuss how they scale with data and compute power. By the end, you’ll have a clearer picture of why these models are at the forefront of machine learning today.

Let’s get started!

Differences between Artificial Neural Networks and Biological Brains

Believing that NNs function similarly to biological neurons can lead to misconceptions about AI. For instance, it might make you think that artificial general intelligence (AGI) is just around the corner when, in reality, we are still far from achieving that level of complexity. It can also lead to anthropomorphizing AI, attributing human-like qualities and emotions to models that are fundamentally mathematical constructs. Understanding this difference is vital for grasping both the capabilities and limitations of neural networks.

Let’s explore how these models diverge from their biological counterparts.

Historical Inspiration of Neural Networks

Neural networks trace their roots back to a time when researchers sought to understand the brain's workings. In 1943, Warren McCulloch and Walter Pitts published a groundbreaking paper titled "A Logical Calculus of the Ideas Immanent in Nervous Activity." Their goal was not to create artificial brains but to construct a mathematical model that represented the behaviour of individual neurons. They aimed to explore how simple computational units could lead to complex behaviours, laying the groundwork for what would eventually become neural networks.

This idea sparked interest in connectionism in the early days, which focused on how interconnected simple units could replicate sophisticated cognitive functions. Researchers began to realise that these artificial neurons could work together to produce intricate activity patterns, similar to how biological neurons operate in the brain. However, this is where the biological inspiration largely ends.

As the field progressed, it became clear that while NNs were inspired by biology, they diverged significantly from it. The models developed were not intended to accurately replicate the brain's structure or function. Instead, they evolved into powerful computational tools that utilize mathematical constructs and algorithms, often bearing little resemblance to their biological counterparts. Understanding this distinction is crucial as it helps dispel myths about AI and clarifies what neural networks can and cannot do.

Differences in Structure

Artificial neural networks (ANNs) are constructed from mathematically simple units that perform differentiable computations. Each artificial neuron takes inputs, applies weights, and produces an output through a mathematical function. This process is significantly simpler than what occurs in a biological neuron, which involves complex electrochemical signalling and intricate interactions with other neurons.

When we examine the scale of the brain, the differences become even more pronounced. The human brain contains approximately 86 billion neurons, each forming thousands of connections with other neurons—estimates suggest there are around 100 trillion synapses in total.

In contrast, even the largest neural networks today, like LLaMA 3, operate as directed acyclic graphs with far fewer connections. For example, LLaMA 3 has about 400 billion parameters, which is more directly comparable to the number of inter-neuron connections in the brain than the number of neurons. This figure is nothing compared to the trillions of synapses in human brains.

Moreover, the complexity of brain connections is far more diverse than that seen in ANNs. Biological neurons can form various types of synapses and exhibit different firing patterns and neurotransmitter types, leading to a rich tapestry of connectivity that supports complex cognitive functions. In contrast, ANNs typically rely on a small set of fixed mathematical functions.

Differences in Learning Mechanisms

Most importantly, the learning mechanisms in NNs differ fundamentally from those in the brain.

The human brain learns in a complex and dynamic process. It adapts by forming new connections between neurons, a phenomenon known as synaptic plasticity. This allows the brain to strengthen or weaken connections based on experience, enabling it to learn from and adapt to new information. Additionally, the brain can create new neurons through a process called neurogenesis, particularly in regions like the hippocampus, which is associated with memory and learning. This lifelong capacity for learning and adaptation is a hallmark of biological intelligence.

In contrast, artificial neural networks (ANNs) operate under a different paradigm. The primary learning mechanism for NNs is backpropagation, which uses a straightforward mathematical operation to adjust weights based on the error of predictions. During training, the network calculates the gradient of the loss function with respect to each weight, allowing it to update those weights in a direction that minimizes error. This process relies on gradient descent, a method that iteratively adjusts weights to find the optimal solution.

However, gradient descent is biologically implausible for several reasons. First, there is no evidence that the brain stores gradients in any form. Instead, learning in biological systems occurs through more nuanced mechanisms involving complex biochemical processes and feedback loops. Second, while the human brain can learn continuously throughout life, modern ANNs typically have a distinct training phase followed by an inference phase. Once trained, these models do not adapt or learn from new data unless they undergo retraining.

Additionally, while some promising research into lifelong learning algorithms and dynamic neural networks aim to mimic the brain's ability to adapt and reorganize, this area remains largely experimental and is nowhere near mainstream application. The neural networks used in practical AI systems today are predominantly static; they do not create new neurons or modify their structure based on experience. This static nature further underscores the limitations of current neural networks compared to the brain's remarkable capacity for continuous learning and adaptation.

Some Necessary Nuances

Convolutional Neural Networks (CNNs) provide an interesting case of biological inspiration in artificial intelligence. While they are indeed inspired by the architecture and essential functions of biological vision, it's important to note that CNNs are not fully biologically plausible, nor do they serve as accurate simulations of the brain's visual processing systems. Instead, they represent a pragmatic approach to leveraging insights from the natural world while optimizing for computational efficiency.

CNNs mimic certain aspects of the visual cortex, particularly in processing spatial information. They employ local connectivity and shared weights, allowing them to detect patterns and features in images effectively. This design draws from how biological neurons respond to localized areas of visual stimuli, making CNNs well-suited for tasks like image recognition. However, while they capture some essential characteristics of biological vision, they do not replicate the full complexity or diversity of neural architectures in the brain.

The key takeaway is that CNNs exemplify how we can take valuable inspirations from biology and apply them to create effective computational systems. By focusing on what is useful for solving specific tasks—like recognizing images or classifying objects—researchers can forgo unnecessary biological complexities that do not translate into computational advantages. This pragmatic approach allows us to harness the strengths of both biological insights and modern computing capabilities, paving the way for advancements in AI while acknowledging the limitations of current models compared to their biological counterparts.

Conclusions

Believing neural networks are direct analogs of biological systems can lead to dangerous myths and misconceptions. This misunderstanding can create unrealistic expectations about what AI can achieve and foster a false sense of security regarding its capabilities.

For example, one common myth is that neural networks are on the verge of achieving artificial general intelligence (AGI), which is far from the truth. In reality, the complexities of human cognition are not just a matter of scaling up current models; they involve intricate processes that we have yet to fully understand.

Another misconception is that neural networks learn and adapt like humans. While they can be trained on vast amounts of data, they cannot learn continuously throughout their lifetime or create new connections like the brain does. This static nature means that once trained, most neural networks cannot adapt to new information without retraining.

Additionally, there is a tendency to anthropomorphize AI systems, attributing them human-like qualities. This misrepresents their capabilities and obscures the ethical implications of deploying such technology in society.

As we continue to explore the powerful potential of neural networks in modern AI, it’s essential to approach these technologies with a clear understanding of their limitations and differences from biological systems. If you're interested in learning more about why neural networks are so powerful and prevalent in today's AI landscape, stay tuned for the follow-up article!

An early version of this article was published in the now defunct Tech Writers Stack. This version has recent statistics, some additional experience, and a whole new section on tooling and workflow.

I decided to try the paid subscriptions model about a couple of years ago. It was not because I thought I could make technical writing my day job—I already have three of those, and I can't imagine not doing any of them. It was mostly for validation: to see if I could get anyone to care enough about my writing to be willing to pay something, not much, for it. And also, it couldn't hurt, right?

Long story short, it worked... somewhat. Regarding validation, I'm more than happy for my 70 paid subscribers. I couldn't imagine even a single person would be willing to support my writing. That extra energy also helped motivate me to write more and, I hope, better.

However, in terms of affordability, technical writing still is little more than a hobby—and sometimes a rather expensive hobby —for me and probably for 95% of my fellow writers here at Substack.

Many of you have deep expertise in areas where you could charge a substantial amount by the hour, yet you choose to share that expertise for free.

And I get it, believe me. I also struggle with the dichotomy of wanting to share as much as possible with the widest audience I can and the reality of having too little time and too much work—like, real work, the one that pays the bill—to be able to give my Substack all the love I'd like to pour into it.

So, over the past year I've tried different models to strike the right balance for doing something that I love—sharing some hard-earned lessons on sometimes fairly niche and complex technical topics—in a way I can afford to.

On one extreme, I tried doing exclusive articles, but I found out some of my best writing was hiding in plain sight from the vast majority of my readers. Why write exclusively for 70 people when other 2000 are eager to read it?

On the other extreme, I tried the patron-like support-me-if-you-can model, but most readers can't afford to support you, even if they want to. And to top it all, the subscription model seems to be declining—I know I have subscription fatigue, and probably you, too.

But I think I've found a sweet spot, and I want to tell you about it because I honestly think if it works for me, it'll work for many of you, probably even better than for me.

The gist is this: write a technical book alongside your Substack, using free articles as the primary material for content and the main channel to drive sales.

Over the last eight months or so, I've experimented with this idea and found a reassuring, if modest, level of success. So, in this article, I want to do three things. First, convince you that this might be a very good business model for you, too, regardless of your background. Second, give you some statistics from my own experience to show you how this might work. Third, offer you some concrete pointers to get started right away.

Buckle up!

Subscribe now

The Big Picture

Here is (just one way) how to do it.

1. Pick a topic you're reasonably knowledgeable about and think you'd enjoy writing consistently about—for example, graph theory is one of those topics for me; large language models is another. Aim to write between 2 and 4 articles a month on that topic.

2. Each article is published for free in your Substack and becomes a new chapter (or part of a larger chapter) in your in-progress book.

   
   Now, here's the part no one tells you (or at least no one told me!) You don't have to wait for the book to be complete before selling it. On the contrary, start pitching the book from day zero.
   

3. Write a launch article with a table of contents or a couple of introductory chapters. Explain to your readers why this is the best way to support your Substack, although they will still get all the content for free. Find a platform to sell PDFs (more on this later) and set a camp there.

4. Add a call to action for every new article to remind readers the book is a thing. Maybe throw in a temporary discount. Also, as new articles become new chapters, you’ll find yourself returning to old chapters and doing some editing, refactoring, etc. This is ok, but don’t overdo it.

5. From time to time, publish a short book progress update article restating the book's purpose and timeline and laying out what has been added since the last update (I do it every four articles or so). This is your chance to reintroduce the whole book idea to potential new readers and remind old readers they can support you that way in more than a footnote.

6. When the book’s content is finished (after no less than a year probably), and a first draft is ready for publication, write a summary article and explain your readers where they can get the finished version (I will most likely self-publish, but you can choose your own adventure).

And that’s it! It's so simple that it seems it can't work, but my early experiments say it works one order of magnitude better than paid subscriptions.

Will anyone get into this?

Yes, believe it: some people will buy the crappy version 0.1, assuming it's an exciting topic! Others will buy the 0.2 and 0.3, all the way to those who will wait until the final draft is ready.

To sweeten the deal, I often set a reasonably low “early access” price compared to what a full technical book might cost. I also offer other perks, such as a private Discord for early readers, a special shout-out in the acknowledgment section, and other things to encourage readers to become part of the process rather than just recipients of the final product.

But mostly, the selling point is just the fact that I'm writing this thing in public, incorporating early feedback. That, and the fact that your readers are amazing and will do everything they can to support you. This gives them something concrete, a one-time transaction that doesn't trigger our dread for another subscription.

Since I stumbled upon this idea, I've completely changed my Substack business model. No more paywalled articles. With this model, I've been able to make all my content open and free while still having something valuable enough that some readers are willing to pay for.

I still have subscriptions enabled on my Substack, though they have now become a free-ride ticket to all my current and future books. Readers who want to support all of my work and can afford to do so will still go for a traditional subscription. But those who can’t or don’t want to be attached to another subscription have this alternative. It's a win-win.

Besides all the advantages in terms of profitability, there's another way in which this pivot has positively impacted my writing, and it's even more important: I'm more motivated than ever to write.

Since all my writing is now public, I know I'll have as many eyes as I could wish for every article. And at the same time, I know I'm not writing exactly for free. Each article brings a trickle of new book sales, not enough to make a living out of it (yet), but enough to trick my primate dopamine-seeking brain.

Some results of my own

If you’re here for the numbers, let me share some of my statistics for the past 8 months. I started with this idea in late February, and I have been running three book projects since in my Gumroad account:

1. The Science of Computation is a long-term project that aims to map all of Computer Science in an accessible and exciting way. It’s perhaps 20% completed.

2. The Hitchhikers Guide to Graph is a fun introduction to graph theory and algorithms that doesn’t require college-level math. It’s about a third of the way, although behind schedule according to my initial intentions.

3. How to Train your Chatbot is a practical handbook on using large language models to build cool applications. It’s almost finished now, and it’s by far my most successful writing project to date.

Each has the same business model. You can buy the early access version for a reasonable fee or a bundle that gets you all three for the price of one. That is my most sold product, of course. I often offer up to 50% discounts as CTAs on new articles, which are the main drivers for new sales.

I have almost 200 sales so far, which is not much by any stretch but is at least two to three times larger than my subscription-based income (exact numbers are hard to come by because of all the middlemen in both models). This is still insufficient to replace any of my regular income streams, but it’s getting closer by the day.

But, but…

"But wait—I hear you ask—writing a book is supposed to be very hard!"

Yes, it is. It is harder than just writing a sequence of more or less interconnected articles. There is definitely extra work involved in writing, formatting, and setting up yet another platform (because why the hell Substack doesn't have that feature yet, right?).

But here is the thing. Writing is way harder than any of that, and you're already doing the bulk of it. The rest is not that much harder. If you can consistently pump one article every couple of weeks, believe me, you can write a technical book.

"Ok but, what if I don't write about a single, focused topic?"

I hear you, I'm writing not one but three (!) distinct books on very different topics and for different audiences because I have many diverging interests—and I have three or four more that I just want to get started on, but I’m sensible enough to understand I can’t handle more than 3 at a time.

Perhaps you can find a subset of your articles that can be tied together into a coherent topic like I did. Or perhaps variety is your thing! Maybe your book is about 20 weird, interesting things few people know (I’m looking at all you polymaths out there!)

"But—I hear you again—I just want to share my knowledge freely!"

I get it, and it's commendable, truly. If you can afford to, that is. But what I've found out is that more often than not, even those who already are in a very stable financial position eventually discover they can't write for free as much as they'd want to. There is a finite amount of time and so many things to do with it. This is a way to make that free writing at least a bit self-supporting.

But you can still make 100% of your content free. That's what I'm doing with all my books. Everything, including source code, is available for free online. The book is just a convenient packaging with some additional syrup and a couple of cherries on the top—if you can pardon the lousy metaphor.

The point is that you can adapt this model to your interests and values. You can make all or most of the content free, or you can make the book an extended version of your articles. The core of the idea is that 80% or more of the work is already done when you hit publish on Substack. And the remaining 20% could be enough to make the whole thing, if not profitable, at least affordable.

Some final words

Listen, I'm advocating strongly for this business model because I'm rooting for you. I truly believe most of you technical folks out there have what it takes to make a compelling book on your area of expertise. But I understand this is not for everyone. So here are some reasons why you might not want to do this.

First, it does take some extra effort. The hardest part is, as you might suspect, consistent writing. I waited more than a year to ensure I'd found the workflow and rhythm that allowed me to write more or less consistently without the rest of my world breaking apart. If you're only a couple of months into technical writing, maybe it's better to wait until you've found your pace.

Second, it does put some extra pressure on you. Once you commit to a topic, you'll have to find a way to stay motivated to write about that same topic for over a year--or as long as it takes to finish the book. You can take some rest here and there, of course. But once you have people paying in advance for something you'll deliver in the near future, some pressure will inevitably build up.

Finally, we all know that once a hobby becomes a job, it stops being fun. There is the risk that this model turns your otherwise pleasurable moments of exploration into yet another thing you have to do instead of a thing you want to do. I don't have any magic potions to fight that. Still, I've found that keeping a healthy variety of topics to write about lets me procrastinate from one topic while investing in another, tricking my primate brain into thinking this is not work.

But all in all, I think this is one of the most underrated business models for technical writers, and there has never been a better time to do it. You have the technology, the platform, and now a community of like-minded writers to support you and help you stay motivated, and sometimes kick you in the butt if we must.

Please go ahead and give this idea at least a fair chance; you'll be surprised by how much you can enjoy it. If you want a bit of advice on getting started, feel free to reach out in the comments and ask around. I’m happy to share whatever little experience I have on this journey.

Leave a comment

You are reading Mostly Harmless AI, a section of Mostly Harmless Ideas where we dive into artificial intelligence and machine learning theory, techniques, and practice. Subscribe for free to get all future issues in your inbox.

Subscribe now

This article is a part of my book on LLMs, How to Train your Chatbot, packed with intuitive explanations, practical advice, and fully working applications using large language models for all sorts of cool stuff. Get it on early access at 50% discount.

How to Train your Chatbot - 50% off

Base LLMs, especially the largest ones, are often all you need. They have proven to be remarkably effective just with well-crafted prompts. However, there are instances where relying solely on base models, no matter how strong, may not suffice. Sometimes, you actually need to train your own chatbot.

Two examples are when you want to incorporate private knowledge or customize a model's behavior to align with specific industry requirements. If the task you need to solve is very specific or the requirements are very strict, it may be too hard to tame a large base model purely using prompt engineering.

In these cases, you can fine-tune an existing model, effectively enhancing its performance for specialized tasks and ensuring it meets your unique needs. This can range from a full training batch to using very efficient and laser-focused tweaks in specific parts of a model.

This article will explore the most important fine-tuning techniques available today, from traditional methods to more parameter-efficient approaches. Additionally, I will give you some practical tips to help you get the most out of your fine-tuning efforts.

Buckle up!

Why Fine-Tuning?

In short, fine-tuning allows you to adapt pre-trained models to domains, tasks, or general requirements for which they were not designed initially with minimum effort. This is achieved with additional training, but some clever techniques are used to make it as efficient as possible. Let's look at some examples of cases where you may want to fine-tune a model.

Solving a Novel Task or Domain

As the landscape of machine learning evolves, new tasks frequently emerge that require specialized understanding and capabilities. Fine-tuning enables existing models to be tailored to these novel tasks without starting from scratch. For instance, a general-purpose language model can be fine-tuned for specialized applications such as legal document analysis, medical diagnosis, or customer sentiment analysis.

Training a model from the ground up is often resource-intensive, requiring substantial computational power, time, and large amounts of labeled data. Fine-tuning offers a more efficient alternative by building upon the knowledge already embedded in pre-trained models.

Since these models have already learned general language patterns and structures during their initial training phase, fine-tuning can often focus on optimizing only a fraction of the parameters relevant to the new task.

Reusing Large Models

The ability to reuse pre-trained models across different downstream tasks is one of the most significant advantages of fine-tuning. Instead of developing separate models for each task, organizations can maintain a single base model that serves multiple purposes through fine-tuning.

That is, you can have a large model trained on a large, general purpose corpora, and a bunch of small “adapter” models that you can plugin to steer the big model as necessary.

For example, a base LLM trained on general text can be fine-tuned for tasks such as document classification or question-answering in the same domain by adding small adapters for each subtask. This means instead of dozens of big models—one for each task—you can have one even bigger model with dozens of small pluggable parts, maximizing the utility of existing resources and reducing redundancy. This not only streamlines development processes but also fosters consistency across applications.

Incorporating User Preferences

User preferences and requirements can evolve over time, requiring adjustments to how models respond or behave. Fine-tuning provides a mechanism to incorporate these changes incrementally without retraining from scratch. By training on new datasets that reflect updated user preferences or feedback, organizations can enhance user satisfaction and engagement with their applications.

For instance, a conversational AI can be fine-tuned to align more closely with specific customer service protocols or corporate idioms or styles. This responsiveness not only improves the model's relevance but also fosters trust and loyalty among users.

Reducing Costs

In many cases, organizations may seek to reduce costs associated with deploying large models due to infrastructure limitations or budget constraints. Fine-tuning allows you to adapt smaller models to achieve performance levels comparable to larger base models.

By hyper-focusing on specific tasks—and forgetting general-purpose linguistic capacities you may not need, such as the ability to answer Wikipedia-like trivia questions—smaller models can be optimized to deliver high-quality outputs without incurring the expenses associated with larger counterparts. This approach enables organizations to maintain competitive performance while minimizing operational costs.

Flavors of Fine-Tuning

Fine-tuning techniques can be categorized based on their computational cost and efficiency. Here’s an overview of the main fine-tuning strategies, sorted from the most costly to the most efficient.

1. Full Parameter Fine-Tuning

In full parameter fine-tuning, all parameters of the pre-trained model are updated during the training process. This approach allows for maximum flexibility and potential performance improvement on specific tasks, as the model can fully adapt to new data. However, it is computationally expensive and requires significant memory resources, making it impractical for very large models or when working with limited hardware.

2. Partial Fine-Tuning

Partial fine-tuning involves updating only a subset of the model's parameters while keeping others frozen. Typically, this method focuses on the upper layers of the model, which are more task-specific, while lower layers remain unchanged. This approach strikes a balance between performance and resource efficiency, allowing for faster training times and lower memory requirements compared to full fine-tuning.

3. Parameter-Efficient Fine-Tuning

Parameter-efficient fine-tuning techniques, such as LoRA (Low-Rank Adaptation) and adapters, involve modifying only a small number of additional parameters while freezing most of the pre-trained model's weights. This drastically reduces computational costs and storage requirements. PEFT methods maintain comparable performance to full fine-tuning but are much more efficient, making them suitable for low-resource environments and enabling easier deployment across multiple tasks.

4. Prompt & Prefix Tuning

Prompt tuning involves adding trainable prompt embeddings to the input data rather than modifying model parameters directly. This technique allows models to adapt to new tasks by optimizing these prompts while keeping the rest of the model frozen. It is a lightweight approach that requires significantly fewer resources than traditional fine-tuning methods.

Similarly, prefix tuning adds trainable tensors to each transformer block in a model. These tensors act as context that guides the model's output without altering its core parameters. Prefix tuning is efficient and effective for certain applications but may not achieve the same level of performance as more comprehensive fine-tuning methods.

Tips for Effective Fine-Tuning

Fine-tuning can significantly enhance the performance of language models, but to maximize its effectiveness, practitioners should consider several key strategies. As usual, it is critical not to miss the forest for the trees and not fall prey to early optimization. Here are essential tips for effective fine-tuning.

1. Exhaust Prompt Engineering

This is the most important advice and the most critical mistake I see small organizations making every single time. Before even thinking about fine-tuning, ensure you have thoroughly explored prompt engineering.

If a state-of-the-art model like GPT-4 cannot solve your task with a well-structured prompt and perhaps some few-shot examples, fine-tuning a smaller model is unlikely to yield better results. Effective prompt engineering can often resolve issues without requiring extensive fine-tuning, which is always more costly.

2. Quality Trumps Quantity

If you have already decided fine-tuning is the way to go, prioritize data quality. Focus on gathering high-quality, diverse examples accurately representing the new task or domain. However much data you can gather will be minuscule compared to the base training set, so quality is the only thing you can control.

Keep in mind that while you can leverage large models with clever prompt engineering to synthesize additional training data, you must always validate these examples with human experts to ensure their relevance and correctness. One great novel example beats 100 excellent but similar ones.

3. Start Small, Grow As Needed

Begin your fine-tuning efforts with parameter-efficient techniques, such as adapters or Low-Rank Adaptation (LoRA). These methods require fewer resources and are often easier to implement than full parameter tuning.

You’ll find tons of open-source implementations of efficient fine-tuning methods, so don’t let the prospect of technical difficulty scare you. Likewise, several LLM providers will let you fine-tune comercial or open source models on their infrastructure, effectively bypassing all need for self-hosting models. This way you get the best of both worlds: a model just for you that someone else takes care of.

If and only if parameter-efficient fine-tuning does not meet your needs, then consider transitioning to full parameter tuning as a subsequent step, but unless you're swimming in private data—and I mean, terabytes of data—you’re most likely safely on the efficient fine-tuning side.

4. Stay in the Loop

Finally, remember the field of AI is evolving extremely fast, with new models and techniques appearing basically every single week. A task that may require fine-tuning today could potentially be solved through effective prompting in next-generation models released tomorrow.

Therefore, avoid over-investing in ad-hoc fine-tuning pipelines that may become obsolete in a couple of months. Also, assess the capabilities of newer models regularly and be prepared to pivot away from fine-tuned proprietary models as soon as a good prompt shows marginally better results. Prompts are far more portable than models.

Conclusion

Fine-tuning is an excellent strategy for small organizations tackling domain-specific tasks effectively. As AI continues to evolve, the ability to adapt large pre-trained models to meet unique business needs is no longer a luxury but a necessity. For smaller organizations and individual developers, fine-tuning offers a pathway to harness the power of advanced language models without the prohibitive costs associated with training from scratch.

Also, leveraging fine-tuning allows small organizations to capitalize on their private data, tailoring models to reflect their specific industry requirements and user preferences. The moat is no longer having the bigger model since some of the best models are open source. The moat is your private data, and fine-tuning lets you get the best combo: a world-class base model trained on privileged data none of your competitors have.

Moreover, in a competitive landscape where larger organizations like Microsoft, Google, and Meta rely solely on base models, fine-tuning gives small businesses an advantage in niche domains. By optimizing existing models with proprietary data, they can differentiate themselves through improved performance and specialized capabilities that address the specific market needs.

And that’s all for today’s issue. Remember you can get my LLM book at a 50% discount and discover tons of practical hints and dozens of working projects to take the best out of large language models.

How to Train your Chatbot - 50% off

Technical writing is one of the most valuable skills you can develop. It is the sort of writing meant to inform, explain, or discuss a concrete, technical topic in a given domain of expertise. Technical writing is factual, logical, and often backed by evidence. It is also terse and devoid of unnecessary adornments, but it doesn’t need to be dead boring.

The main purpose of technical writing is to produce in the reader a new level of understanding —as opposed to, say, creative writing, whose main purpose might be to entertain or inspire, or political writing, whose main purpose might be to make the reader sympathetic to some cause. Almost everything you read in this blog, including this post, is an example of technical writing.

Since technical writing is about understanding, writing should focus on truthfulness and clarity, while being inspiring and engaging. In this post, I’ll show you a loosely defined process I follow to achieve those objectives, making it easier to write often.

This article is an updated and improved version of an old article I published in when I was just getting started with technical writing. This new version incorporates the experience of more than 100 articles written and published since, which have garnered more than 50 thousand views in total.

A Framework for Technical Writing

In this article, I will outline the CODER framework for technical writing: a structured approach to writing that I’ve converged on, designed to ensure clarity, coherence, and quality. Each step builds upon the previous one, guiding authors from idea collection to final publication.

The core of this process consists of five steps:

1. Collect: Gather all ideas, thoughts, talking points, arguments, evidence, etc.

2. Outline: Organize those ideas and create a structured outline of the document.

3. Draft: Write, expanding on all the ideas and filling in any gaps.

4. Edit: Improve the structure, content, and style of the document.

5. Release: Finalize the article, including formatting, adding images, tags, and metadata, and preparing it for publication or distribution.

In the rest of this post, I will explain why this process is useful, then I will detail each of these steps, and finally, I will give you some concrete advice on improving the quality of the content, the clarity, and the style of your writing.

Why technical writing is different

Technical writing is a form of communication that explains complex topics clearly and concisely. As in any form of writing, there are basic linguistic and stylistic considerations that you should follow, the most important being clearly defining your target audience. Only once you know who you’re writing for can you decide your content's tone, style, and depth.

However, there is one way in which technical writing is special. As I said in the introduction, technical writing aims to produce a form of understanding in the reader. This means there is something you know, and you want to transport that knowledge into your reader’s mind. The problem is that the things you know are not separated in your brain, and they are inside a single file that can be downloaded. No, they are interconnected with everything you know.

Thus, technical writing involves taking an amorphous web of interconnected ideas inside your brain and transforming it into a neat package that can be delivered with laser precision to your reader’s mind to become an amorphous web of ideas interconnected inside their brain. Like a pill, technical writing consists of a core set of clearly defined ideas wrapped in a nicely digestible envelope.

Types of technical writing

Technical writing encompasses three main types: informative, didactic, and argumentative. Informative writing focuses on providing factual information on a given topic. Didactic writing is more focused and instructional, usually providing step-by-step guidance or describing the process of doing something. Argumentative writing is slightly different in that it uses facts and evidence to make a case to try to convince readers of a particular opinion.

Some details of the writing process will change depending on the type of article you’re planning, but the general workflow I explain in this post works for any technical article.

Informative articles are designed to impart factual information without any bias from the author, allowing readers to form their own opinions. This type of article covers various topics, from news stories and current events to historical accounts. By presenting facts in an unbiased fashion, these articles help readers to make informed decisions and develop their perspectives.

Didactic articles are designed to help readers learn something new. They can cover a wide range of topics, from deeply technical algorithms to broader technologies. The key characteristic of a didactic article is its ability to present complex information in an organized, effective way. This often involves providing examples and visualizations to help the reader understand the material, as well as utilizing analogies and intuitive explanations to make the material more accessible.

Argumentative articles are the most opinionated and persuasive pieces of writing; they are designed to convince readers of a certain point of view and to argue in favor of a specific opinion or against another opinion. To achieve this, argumentative articles are usually written as essays containing logical arguments presented in a way that is easy to understand. The arguments must have a clear structure, each step building upon the previous one to establish a solid foundation.

Choosing what to write about

All writing starts with choosing a topic. In the case of technical writing, that topic is always something you understand better than your reader. But it doesn’t necessarily need to be a topic you’re an expert on. It can be something you just learned or something you’re working on.

An idea for finding a technical topic you feel comfortable writing about is visualizing your reader as the person you were one or two years ago. Find a topic you can explain to a one-year-ago version of yourself that would make them understand it faster or easier than what it cost you.

In my case, I often write with my undergrad students in mind. I chose topics I think I would have been interested in reading about as a student. You can also do market research: find what your readers consume online, make polls, or ask a few of them directly.

The act of choosing a topic is an exercise in defining your audience. You always write for somebody, so the better you can profile that person, the easier it will be to find an interesting topic to tell them about. When you do that, you should be able to write a single sentence that conveys your primary objective with this piece of writing. In the case of this post, it would be something like this:

Help beginner technical writers improve their process with a workflow that will quickly transform a loosely defined idea into a polished article that is coherent and interesting.

Dissecting the CODER Framework

The workflow I will describe in this post is based on distilling what you know down to its core meaning and packaging it in an optimally digestible way. To achieve that, the key principle is to disentangle content from style.

First, you will define exactly what you want to say and in which order, devoid of any stylistic considerations. That is the core. Then, you’ll massage, enrich, and polish it, growing an envelope that is easy to digest.

Let’s go.

Step 1: Collect

Begin with an abstract objective for your article and a clearly defined audience. The first step is to expand that abstract idea into concrete talking points: the core ideas that you think are necessary to understand the topic.

For now, you only want to produce a set of short sentences for each core talking point. You don’t want to focus on anything related to the style of the writing; you just want to get all the ideas that could be relevant. Do not bother about order or structure. There is time for that later.

Keep in mind that these are claims, not merely objectives. Most people approach this stage by listing questions they want to answer, like “Explain why technical writing matters”. The problem with this approach is that you’re just delaying the actual task of getting those answers out of your brain. Instead, you should write your claims as explicit answers to those questions: “Technical writing is useful for understanding a topic better because it forces you to find sound arguments”.

You can start by listing questions, but then you must answer them. I've found that starting with some loose questions helps mainly when writing a didactic or informative piece. However, when I'm writing more opinionated essays, I often already have some concrete claims that I want to inject into the reader.

The reverse outline is the best technique for producing a solid core set of ideas. This works by distilling your mind from an unstructured, raw dump of it down to concrete talking points or claims. You can do this by just letting your mind run free and writing everything that comes to you about that topic.

But it’s much easier to talk than write, so I’ll record myself talking for 10-20 minutes on a particular topic, speaking my thoughts aloud without worrying about order or structure. It works best if I take a walk and use a recorder app on my phone. After I record the audio, I will transcribe it and go over each paragraph to distill it down to its central idea.

At the end of this step, you should have a comprehensive list of core ideas, supporting arguments, evidence, etc. This is all you could talk about.

Step 2: Outline

At this point, you have a set of more-or-less independent claims with all their support. Once the central claims and arguments are in place, it’s time to give some structure to the whole piece. Only now will you consider sections, subsections, and paragraphs.

Depending on what you want your reader to focus on, you will order the ideas differently. For example, if it’s a didactic piece, you will probably want to begin with intuitions that help the reader understand the why before the how. If it’s a more informative piece (e.g., a historical account or a summary of recent events), you’ll probably want chronological order. If it’s an argumentative piece, you’ll want to sort the claims so each one is supported by previous claims.

At this point, you may find that some of your claims are not sufficiently developed or missing a link somewhere. This might prompt you to return to step 1 and refine your claims. These first two steps can thus be somewhat cyclic. You can collect and outline, just to notice that you missed some key ideas, and then collect and outline again. Just don’t overdo it.

At the end of this step, you should have a clear picture of the final document layout, sections, subsections, core claims, and supporting arguments in their optimal order.

Step 3: Draft

At this stage, the ideas have all been incorporated into an organized structure effectively conveying a message. Now, each claim, argument, or point must be fully fleshed out into a comprehensive paragraph. This means elaborating on the points with examples, explanations, and additional ideas and ensuring the sentences and ideas flow together cohesively. The audience plays a crucial role here. How much they know about the subject will influence how deeply you dive.

Just write. Prioritize quantity over quality.

Don't become too fixated on adhering to one particular style right now. Instead, focus on developing your ideas and getting them across. Don't be afraid of using phrases that seem superfluous, as they can often help keep the writing flowing. Don't worry about how the sentences and paragraphs are laid out; you can refine them later. Don’t worry about verbosity and repetition either; you will come back, tweak, and polish your writing until it is exactly what you want.

Reusing content from step one of the process can be a useful way to quickly fill up space, but often, the structure you come up with after step 2 is so different that it makes more sense to start the process again. One possible way to do this is to record yourself speaking and simply write down everything you said. You could use an AI transcription tool to do it more quickly.

At the end of this step, you should have roughly one-and-a-half to twice the amount of text you need and be satisfied that everything that could be said is in there.

Step 4: Edit

With your draft in hand, it’s time to roll up your sleeves and dive into the editing process. This stage is where the real magic happens, transforming your rough draft into a polished piece of writing. Editing is not just about correcting mistakes; it's about enhancing clarity, coherence, and overall impact.

Editing works better if you use a top-down approach, focusing first on the central issues with the document structure and high-level ideas and then zeroing down on the specific wording of each phrase.

So start by taking a step back and examining the structure of your article. Does the flow make sense? Are your arguments logically sequenced? You can rearrange paragraphs or sections to create a more compelling narrative. Don't hesitate to cut something out if something feels out of place or redundant. Remember, every paragraph should serve a purpose and contribute to your main message.

Next, zoom in on the details. Read through your draft with a critical eye, focusing on grammar, punctuation, and sentence structure. This is where you can refine your language to ensure it’s precise and engaging. Read your work aloud; this technique can help you catch awkward phrasing and identify areas where the rhythm feels off. If a sentence doesn’t flow smoothly, rework it until it does.

Always consider your audience at this stage. Are you using language that resonates with them? If your readers are novices, avoid jargon or provide clear explanations. If they are experts, ensure your content is rich with insights and depth. Tailoring your writing to your audience’s knowledge level is crucial for effective communication.

Don’t shy away from seeking feedback. Share your draft with trusted colleagues or peers who can offer constructive criticism. Fresh perspectives can highlight strengths and weaknesses you might have missed. Be open to their suggestions, but also trust your instincts about your content.

Finally, take a short break before performing a last read-through. This brief distance can help you spot inconsistencies, typos, or awkward phrases.

By the end of this step, your article should be refined and ready for the next phase. You should feel confident that your message is clear, your arguments are compelling, and your writing is engaging. Editing elevates your work to its highest potential, ensuring it resonates with your audience and effectively communicates your ideas.

Step 5: Release

Now that your article has been meticulously edited and polished, it’s time for the final step. This is where you take all your hard work and prepare it for the world to see. The goal here is to ensure that your content is ready for publication and optimized for maximum impact.

Begin by focusing on formatting. Ensure your article adheres to any specific guidelines or standards required by your publishing platform. Check that your headings, subheadings, and fonts are consistent throughout the document. Proper formatting enhances readability and gives your article a professional appearance.

Next, it’s time to add supplementary elements to enhance your content. This includes images, charts, or infographics that illustrate your points and make the article more engaging. Ensure all visuals are high-quality, properly attributed, and relevant to the content. Don’t forget to include alt text for accessibility and SEO purposes.

SEO is a crucial aspect of this final step. If your platform allows it, add relevant tags and metadata to your article to improve its visibility online. Consider keywords your audience might search for and incorporate them naturally into your content. This will help your article reach a wider audience and drive more traffic to your site.

Before hitting that publish button, take a moment to do a final review of the entire article. Check for any last-minute errors or formatting issues that might have slipped through the cracks. Ensure that all links are functional and that your citations are accurate. Read the article from top to bottom on a different device and screen size to where you wrote it, e.g., on your phone. This will help you more easily catch subtle formatting errors.

Once everything is in order, it’s time to publish your article. Publish it on your chosen platform, whether a blog, a technical journal, or a social media site. Share it with your network and promote it through various channels to maximize its reach.

Finally, take a moment to celebrate your accomplishment! Completing an article is no small feat, and you should be proud of your work. Remember, the Release step is not just about publishing; it’s about sharing your knowledge and insights with others. Your article has the potential to inform, educate, and inspire, so embrace the opportunity to make an impact.

Final considerations

When I wrote this post, I went into it with a good sense of the structure I wanted it to follow. Thus, I wrote the introduction almost entirely before moving on to the other sections. I added the first three sections when the rest was almost ready because I felt an introduction was needed. However, I still made sure to continually iterate on the ideas and arguments within each part of the document. I strived to ensure that each one was clear and concise and that I was expressing my thoughts most effectively.

After step 3, probably not all sections will evolve simultaneously. You might have sections that grow faster and arrive at style check, while others are still barebone bullet points. That’s fine, too.

This workflow can help you write better, avoid writer’s block, and make a pleasure out of writing. But the end goal is getting the article out there, not the process right. Anything that gets in the way of writing is an obstacle, so feel free to deviate. It's okay to adjust it to fit your writing style or any specific needs. Try to be flexible and have an open mind when approaching your writing workflow, as this will help you to create a positive experience.

Asking for feedback

You can ask for feedback at any step of the process, as long as you ask for the exact kind of useful feedback, although most people are only comfortable asking for feedback after the first round of edits.

In part, writers do this because many readers won’t be able to disentangle content from style, at least on a first read, so if you ask them for feedback on the initial unstructured list of claims, they will probably flood you with useless —at this point— suggestions about word choice, grammar, and sentence structure.

The solution to this is two-fold.

First, ask for specific feedback. Instead of “What do you think about this?”, ask specifically about your step. “Hey, I’m writing a piece on X, here are the main talking points I plan to cover… Do you see anything missing?”. Or, when you’re in edit mode, you can ask, “What is the most boring 20%? What is the 10% that I must keep?”.

The second way is to be selective about who you ask. Some people are better at analyzing factual content, and others are better at proofreading or style checking. Don’t ask for the same feedback from the same 100 people since you won’t be able to read it anyway.

And finally, understand that all feedback is optional, and only you know what message you want to convey. You should pay attention to your readers’ suggestions because they are what you write for but, if advice goes against your gut feeling, trust your instincts.

A note on writing tools

You may have noticed there’s nothing in this guide about how to implement something like this workflow. It is purely conceptual. You could do it with pen and paper, a traditional word processor, a specialized writing app, or an AI-powered writing assistant. That said, some parts of this process can be aided with the right tools.

A good text editor will let you move entire blocks of text quickly, navigate back and forth between sections without having to scroll with your mouse, fold and hide pieces of text when necessary, check grammar and orthography, and add comments that won’t be shown in the final version to guide yourself. All of that will help you a lot.

Likewise, you can speed up a lot by talking and transcribing instead of directly writing. You can transcribe manually, but there are plenty of good automatic solutions out there, some even free.

Regarding generative AI tools, there are, of course, many takes on what is a genuine and ethical use of them to enhance but not replace your voice. You can use them well without losing your voice or being disingenuous. This article has seen maybe 30% of its content go through AI writing tools in all stages, from idea collection and outlining to the final tweaks, but I’ve reviewed every single word and tweaked almost every sentence afterward.

I use generative AI to rewrite sentences more clearly, expand on some talking points, suggest claims or ideas similar to the ones I already have, transcribe my audio recordings, extract essential claims from longer paragraphs, etc. As long as you are conscious of the process and stand by the final text, I would call that a fair use.

Recap

Here’s a quick recap of the whole workflow.

You start with a nebulous idea that you need to distill into concrete talking points. Then, you will review, refine, and support those talking points with evidence, arguments, and citations until satisfied they stand independently. Next, you will sort and cluster them to thread an outline that connects all the dots and fills in the gaps.

Then, you write as much as you can without worrying about quality. This is your first draft. Now you become your harshest critic and carefully edit your article, but don’t overdo it: often two passes—one for high-level, structural review and one for wording and style—are enough. And finally, you release.

The main takeaway of this article is to separate content from style as much as possible. Decide what you want to say first and then how you want to say it. That’s the key to high-quality and scalable technical writing. The rest is up to you.