Today I want to tell you a different kind of story. It’s not about machine learning, large language models, algorithms, or theory of computer science.

It’s about a side project that I’ve been building for a couple of weeks that made me fall in love again with an ancient love of mine. The quick and easy way to explain it is this: a performance-focused graphics engine for data-driven visualizations in Python.

But that’s only the surface. If you want to see the coold demos and the technical description, feel free to scroll down. But if you want to know the story behind it, let me start from the beginning.

Subscribe now

The Origin Story

So, this starts back in undergrad, before I did anything related to machine learning or optimization or statistics. My first love was actually computer graphics. I had “learned to code” like, I don’t know, at 11 or 12, and for the first five years or so, before getting to college and actually learning to code, all my “coding” was basically tiny games. It was RPG Maker back then—who remembers that?

I always wanted to be a game developer, as you may imagine, and I think that’s probably the main motivation why I studied Computer Science. There are two kind of people who want to study Computer Science, as a matter of fact. One is people who love games—and the other is, of course, people who hate games; there are no in-betweens.

I was the loving-games kind and all I wanted to build games for a life. So when I was in first year, after actually learning some real coding, my first kind of large project was a game engine. This was before Unity, before even XNA—who remembers that?—this was when .NET was getting started, and I wrote a quick and dirty game engine in pure C# that talked native DirectX 11.

It was very cheap, a disaster of architecture almost surely, but it taught me the basics of how to construct a scene graph, how to animate a camera, how to do lighting, how to write very basic shaders. I learned a ton and basically fell in love with computer graphics.

I ended up doing my diploma thesis in computer graphics—screen-based global illumination, a couple of years before NVIDIA came up with ray tracing on the GPU, which basically killed that whole area of research. And I also did my Master’s on global illumination and some data structures for the GPU, but after graduation I quickly switched research towards machine learning and AI, which, you can imagine, this 2014, and deep learning was just on the rise. The rest is history, as they say.

And then, here I was this past week thinking about old projects that I used to have fun with when I was in college, and trying to remember what it felt like to code back then, no LLMs, no internet for the most part even. The time where I’ve been the most fun was probably when I dabbed into procedural generation of cities, mountains, lakes, and… stuff, in the late 2012. This was at the early era of PCG, and I never got to do anything with that other than a few tutorials and a few lessons that I taught at University.

I played with Unity for a couple of years, but nothing too serious–I think I was actually one of the first people in my University to even install Unity, and I even taught a couple of Master’s courses on it. I participated in a couple of game jams, but after 2017 or so I stopped doing graphics all together. And I’ve been doing machine learning since.

But, in any case, I kind of forgot about computer graphics all along. At least during day-worked. So there was I last weeek, remembering that and asking how hard would it be to actually make a graphics engine in Python, some quick hack like my undergrad projects. I did a bit of research and I discovered that Python is, as of 2026, in a very good position to build a graphics engine, and not just a crapy one, but one that is actually fast. We have WGPU now—the spiritual and practical succesor of OpenGL (who remembers that?), which has native suppotr for GPU-accelerated graphics in Linux.

I basically did a plan and sat for three days to hack this thing.

The Engine

My first idea was to have a Rust backend for all the graphics engine stuff—the rendering loop, materials, lights—but I quickly decided to drop that idea because getting Rust and Python to talk to each other was becoming increasingly harder and harder, and I really wanted to finally see a damn cube rendering on my screen.

So I decided to switch completely to Python. But since I’m a grown-up now, I have to find some kind of serious objective for making something like this. I decided I didn’t want to make a typical graphics engine where you have a scene graph with hierarchies of entities and properties, and you simply render all of them. No, that is way too 2000s.

I decided I wanted to do a very fast, data-driven visualization tool purely based on the Entity-Component-System (ECS) paradigm and make it extremely performant, so it would focus on big data-driven simulations like N-body simulations, chemical and physics experiments, AI pathfinding and agents, you know, grown-up stuff like that.

(But actually, all I wanted was to play with WGPU and draw some cubes in Python. Wink, wink.)

This framing gave me two things, though. My solution doesn’t have to be very fancy as a game engine, we don’t need to be able to like load skeletal animations or stuff like that. It’s not actually a game engine; it’s a graphics engine with at best some interaction logic. But it still lets you do some cool stuff, even if all you can render is blocks and spheres. When you can render thousands of them running very fast on the GPU, you can do some cool stuff. So this is the motivation, and now let me show you what I have.

Deep Dive

So here is manifold—short Manifold Graphics if you want. It’s a Python library built on top of WGPU, a graphics engine based on the Entity-Component-System paradigm.

If you have never heard about it, ECS is a completely different way of writing code that is especially tailored for video games, but it is very little known outside of the game development world. And its awesome.

In a typical business code, you have entities who own their data, and you usually have behavior associated to entities; so entities also own their behavior—this is the basic Object-Oriented Programming paradigm where objects own their data and their methods. And if you want to do something with an object, you have to call methods on the object so the object guarantees the instance invariances.

Since OOP was basically the ONE programming paradigm of the 90s and early 2000s—when the videogame industry really exploded—it is only normal that we started writing games like this. But there is a problem with OOP (well, many problems, but one in particular that matters for our discussion).

When you have 10,000 objects, each of them with more or less the same structure, e.g., they are physical particles bouncing with each other, or little zerlings comming to your base, you simply cannot update them fast enough. For example, making a physics simulation out of this is extremely slow if you have to go to each particle and update its velocity, its scale, its rotation, etc. You’ll end up doing thousands of tiny method calls, thrashing your cache, and issuing lots of super small copies to GPU for drawing.

What you want is to vectorize this operation. You would like to have all of the objects’ data in a single NumPy matrix, and you want to write a very, very efficient vectorized code that doesn’t do any loop and just updates everything at once. THen copy all the data to the GPU and issue a single draw call that renders all objects parameterized by their positions, rotations, etc. Chef kiss.

This is the Entity-Component-System paradigm at its core. It completely flips the responsibilities from standard OOP The components are just flat storage of data (rows in a matrix) and the entities are just pointers to a row where all of their data lives. Then the systems are methods that act on a subset of entities using heavily vectorized code, because each system deals with a large number of equally-structured entities, and they don’t care which is which.

In manifoldx, each system is a Python method that receives a subset of entities that have some combination of components. For example, if you want to process all of the particles in a simulation, you write a system that receives entities that have the Particle component, perhaps also a Transform component. In the transform component, you will have the position, rotation, scale, and the particle component will store simulation-specific data like velocity, temperature, momentum, etc.

The key to high performance in ECS is to avoid looping as much as possible. You assume all of the components of the entities in a system have exactly the same layout, so what you get is really a view of a matrix, and you write vectorized code. You add something to all them, you multiply all them by something, or in general you compute some matrix operations on them. All at once.

And if you can write your code like this, then you get a very, very fast rendering loop because instead of making one method invocation per entity, you make one method invocation per archetype, that is, per combination of components, which is a couple of order of magnitude less that your entities count.

Here’s a minimal example showing how the ECS works in manifoldx:

import manifoldx as mx
import numpy as np

engine = mx.Engine("Cubes")
engine.camera.zoom(0.1)

# These are all static things that are created
# and stored in memory once
mesh = mx.geometry.sphere(1)
material = mx.material.phong(mx.colors.BLUE)

# Custom component, gets registered in engine to keep track
# Only used for reflection on the values
@engine.component
class Particle:
    velocity: mx.Vector3
    angular: mx.Vector3
    life: mx.Float

# This runs every frame
@engine.system
def particle_lifecycle(query: mx.Query[Particle, Transform], dt: float):
    query[Particle].life -= dt  # Single vectorial operation
    query[Transform].position += query[Particle].velocity * dt
    query[Transform].rotation += Transform.rotation(euler=query[Particle].angular * dt)
    query[Transform].scale = query[Particle].life / 10.0

    # Destroy all dead particles at once
    engine.destroy(query[Particle].life <= 0)

    # Now we create lots of particles
    N = int(100 * dt)

    # This will in principle reuse the buffers for dead entities,
    # but will expand the buffer if necessary
    engine.spawn(
        Mesh(mesh),
        Material(material),
        Transform(pos=(0, 0, 0), scale=(1, 1, 1)),
        Particle(
            velocity=np.random.uniform(-5, 5, (N, 3)),
            angular=np.random.uniform(-2, 2, (N, 3)),
            life=np.random.rand(N) * 10,
        ),
        n=N,
    )

    # Update camera
    engine.camera.orbit(45 * dt, 0)

That’s it. A single line of code to update all positions at once. Notice the query argument that defines which entities you get (all entities with both a Transform and a Particle component). And here is how that looks like:

In a real simulation, you can have, say 10 systems, but you have 10,000 or 100,000 entities, and you know you can do very fast vectorized updates in NumPy for all them, 10 times each frame.

For example, if you have 500 particles and you want to do N-body simulation, computing the 500-squared gravity interactions 60 times per second in Python is suicide. But if you do it in NumPy, then you get something that runs in a few milliseconds. A quarter million interactions computed 60 times per second. In Python.

To make it really efficient, you need to also avoid copying or moving data; it’s all masking and clever NumPy layout that keeps all of the memory in one place, and you are just seeing fragments of that memory in each system.

The other key idea is that you don’t modify anything in a system. That line where position is set, doesn’t really write back to the matrix. All it does is compute the right-hand side and then you issue a command that will be run at the end of all the systems, before frame rendering happens. This allows to write pure threaded parallelism, because you can run several systems in different threads—they are all reading the same data, but they aren’t writing to the buffers, which is great since Python has real support for multi-threading now in 2026 (after 35 years!).

Showcase

That is the basic idea. Now lets see some examples. AS of today, version 0.2, manifoldx has some basic shapes like cubes, spheres, and planes, and support for basic PBR lighting, camera controls, and that’s basically it.

All the engine realy does is set up this somewhat clever inversion of logic that forces you to write very efficient code, and the magic is in what you do inside the systems.

So let me show you three examples.

1. N-Body Gravitational Simulation

The first is an N-body simulation. All gravity computation happens in a single NumPy block with no Python loops. The only relevant part of the code is the gravity system, that looks something like this.

@engine.system
def nbody_gravity(query, dt):
    pos = query[Transform].pos.data  # (N, 3)

    # All-pairs position differences: (N, N, 3)
    diff = pos[None, :] - pos[:, None]
    dist = np.linalg.norm(diff, axis=2)

    # Force magnitude: G * m_i * m_j / r²
    force_mag = G * mass_prod / np.maximum(dist, SOFTENING)**2

    # Net force = sum over all other bodies
    net_force = (force_mag[:, :, None] * diff / dist[:, :, None]).sum(axis=1)

    velocities += (net_force / masses[:, None]) * dt
    query[Transform].pos += velocities * dt

This runs 500 bodies with 250,000 force pair computations at 60fps.

2. Ideal Gas Simulation

The second example is an ideal gas with elastic collisions inside a bounding box. Again, all running without a single for loop. Collision detection and impact resolution in vectorizednumpy operations.

@engine.system
def gas_physics(query, dt):
    pos = query[Transform].pos.data

    # Wall collisions: vectorized mask
    below = (pos + velocities * dt) < -BOX_HALF
    above = (pos + velocities * dt) > BOX_HALF

    # Here we avoid branching and use masking instead
    velocities[below] = np.abs(velocities[below]) * RESTITUTION
    velocities[above] = -np.abs(velocities[above]) * RESTITUTION

    # Particle collisions: find overlapping pairs
    diff = pos[None, :] - pos[:, None]
    dist = np.linalg.norm(diff, axis=2)
    overlap = dist < 2 * PARTICLE_RADIUS
    i_idx, j_idx = np.where(np.triu(overlap))

    # Resolve collisions with impulse
    # ... (collision resolution code)
    # ... (also vectorized)

    query[Transform].pos += velocities * dt

Here’s how that looks like.

3. Boids Flocking

The third example is a Boids simulation with emergent flocking behavior. This is the one that strikes me the most because boids simulation is often compute-heavy. Each individual entity must keep track of a subset of neighbors and adjust behavior based on them, not the whole set of entities. But again, a bit of numpy magic lets us vectorize the crap out of this and simulate 300 boids at 60 frames per second.

@engine.system
def boids_physics(query, dt):
    # Separation, alignment, cohesion as vectorized tensor ops
    diff = pos[None, :] - pos[:, None]  # (N, N, 3)
    dist_sq = (diff * diff).sum(axis=2)

    neighbors = dist_sq < PERCEPTION_SQ

    # Separation (1/dist² weighted)
    sep = (-diff * (neighbors[:,:,None] * inv_dsq[:,:,None])).sum(axis=1)

    # Alignment (average neighbor velocity)
    avg_vel = (vel[None,:] * neighbors[:,:,None]).sum(axis=1) / safe_count

    # Cohesion (steer toward center of mass)
    center = (pos[None,:] * neighbors[:,:,None]).sum(axis=1) / safe_count

    # Plus predator avoidance and boundary steering...
    # That one is easy.

Here goes a sneak peek.

You can check all the examples in the Github repository to see the full code, but the bulk of the implementation is these cleverly vectorized system methods.

Future Directions

And that’s it. This is my pure Python (well, you know what I mean) graphics engine for serious, grown-up stuff that is surely, definitely, not a weekend side-project meant to procrastinate on actual work... I mean, what?!

Where I will go with this? I don’t know. I always write these things mostly as a learning exercise and I’ve learned a lot about graphics in Python. I’ve updated my view of modern graphics and I think I’ve paid my debt of the last seven years in graphics computation. I’m kind of happy now that I know how to do this in 2026. Mission accomplished, I guess.

There are some places this engine can go to, like some custom shaders when you need stuff like lighting effects. But it is not going to become a traditional, full-blown game engine. I will not add support for lots of game engine-like features including, I don’t know, skeletal animations, level of detail, scene management, or, god forbids, visual scripting and nonsense like that.

Now two areas I’d like to explore in the future. One is extending the engine towards the kind of behavior you need to write for AI simulations. If you want to run some sort of agent simulation or ant colony optimization or stuff like that, that code doesn’t look that much as a frame-by-frame update, but like an asynchronous event-loop—which is also something that is not usual in game engines. And the other direction is towards procedural generation of meshes and content in general, which is an area I left five or six years ago and would pretty much love to come back to it.

And that’s it for this week. This is not production-ready at all—it’s mostly a toy at the moment—but you can take it apart and hack your way into some cool physical or mathematical simulation. The code is on GitHub if you want to try it yourself, and I’d love to see what you build with it.

Until next week, stay curious.

I know, this is the probably the twentieth article you’ve read this week on “how I am using AI to automate my whole life/work/whatever”. Me too. But please, give me a couple of minutes to tell why this article might be different, and thus interesting to you.

But even if you don’t have two minutes to spare, please check the repository where all I’m going to tell you about is implemented, ready for your taking. Clone it and play with it, then if you like it, come back and read the rationale behind it.

Done? Ok, here we go.

So yes, this is another article trying to explain to you how I use AI coding agents (specifically Gemini CLI, but the specifics don’t matter) to enhance my workflows. Here are a couple of reasons why I think you might be interested—and why this article might be different to so many lookalikes out there.

First, I’m not an enthusiastic techbro who just discovered AI. If you’ve read this blog before, you know I’m a longtime researcher in AI—way before LLMs were a thing—and also a self-proclaimed AI anti-hypist. I’m not just overexcited about this shiny new toy. I’ve been using generative AI since day one for everything, and I’ve been telling you exactly how it sucks at almost anything important since day one. Coding has been the same until very recently. I can tell you the change in productivity is real, provided you are responsible and considerate.

Second, my approach to incorporating AI into my workflows is very careful and grounded in a large dose of healthy skepticism. I know firsthand how these things fail, so my approach attempts to be very robust to hallucinations and context drift and all the plagues of even the most powerful LLMs.

Third, I have a thing for systems. What I’m going to show is not just a set of hacks or clever prompts or productivity tips. It’s a principled system to go from ideation to research and planning to execution at the fastest responsible speed, without sacrificing on safety or maintainability.

And fourth, I have kind of a unique position in that I’m both heavily invested into coding as well as technical writing. I’m a college professor, so I do a lot of research, writing, and editing; but I also run a small AI startup, small enough that I get to do a large part of the coding. So my system attempts to bridge these two facets—code and prose creation—with the same unified principles of careful deliberation and planning, and robust tracking of the project evolution.

If you’re curious already, let me start by exposing the overall principles behind this approach, and then we’ll dive (not delve, but close) into the details.

Subscribe now

Principles of Effective AI-Assisted Work

By far the most pressing limitation of modern, top-tier LLMs for my line of work is context saturation. What I mean by this, is no matter how many tokens your model can fit (Gemini 3 claims to chug as much as 1 million tokens), when you work for a considerable amount of time on a single project, you will exhaust this context. And even if the context window isn’t full, the model will quickly lose the capability to keep track of the important bits of context, and will start to deviate from your instructions and make up its own agenda. Not out of evilness but simple probabilities.

The way you see this problem when using Claude Code, Gemini CLI, Codex, Copilot, or anything similar is twofold. The model will either forget midtask what is was supposed to do, but it won’t simply fail; it will reinterpret the task based on its faulty, lossy, blurry viewpoint given the available context and it will do something close, but not quite what you wanted. Or, the model will make faulty assumptions and forget to ask relevant questions, again behaving close but not quite exactly as you intended.

The result is always the same, you get frustrated that you achieved 95% of what you needed, but the remaining 5% is harder to fix than to just try again. And what could have been a happy working session where you get important and difficult things done quickly derails into a session of arguing with an LLM trying to convince it (him/her/pick your side) to do things the way you want.

Barring any fundamental paradigm shifts in the near future, under the assumption that LLMs will keep working as they are, the only way to fix this is to be very conscious and careful about the context—what has been dubbed context engineering—in two senses: First, do not pollute the context with unnecessary details. And second, re-inject into the context whatever is relevant for any given task, so it doesn’t get forgotten.

I know, kind of contradictory, but tradeoffs are what engineering is all about. In my system, I’ve applied three principles to help me manage these tradeoffs effectively.

1. The important things should be made explicit.

2. Resist the urge to guess.

3. Delegate, delegate, delegate (yeah, three times).

Here’s how that works. For principle one, we will keep track of everything important in markdown files in the repository. This means ideas are committed to plans in markdown files before acting on them, research is summarized and stored in real time, and everything that changes in the project is logged to a long-lived journal, so the model remembers why we made some decision months ago.

For principle two, we will favor using explicit commands that are translated into explicit prompts, instead of relying on implicitly activated skills that you have little control of. So if you want the model to make a plan, you will prompt it with “/plan lets design feature X”, and the /plan command will invoke a carefully crafted prompt that says how plans work, where they are stored, etc.

And principle three means using sub-agents a lot. This is a Gemini CLI specific feature—but every other coding agent has a similar thing—where you can launch a complicated task as a “sub-agent”—which basically means a custom prompt—but here is the important part: All the context of that sub-agent is kept private, not shared with the main agent, so the internal reasoning the sub-agent needed to run to find 20 different sources in Google does not pollute the main context. We only receive back the summarized responses. This allows running very long tasks (my record is a 30 minutes long research loop, involving hundreds of retrieved web pages) on a single agent turn, without exhausting the context.

I use four sub-agents in different commands. The planner is the lead architect. It’s a read-only agent that walks through your codebase and reads everything necessary to understand architecture, design decisions, etc., given a specific task. It then produces a detailed Markdown plan in the plans/ directory—a physical source of truth that you can review before any code is touched, and that the main agent will follow step-by-step. This separation prevents the system from “guessing” its way through your codebase.

When I need external knowledge—like a library’s latest API or a specific technical specification—the researcher agent takes over. It scours the web to fetch relevant documentation, which it then synthesizes into granular summaries in the research/ directory. This raw data is then handed off to the main agent to build an executive report annotated and linked to all relevant sources, again all stored already in your repository.

And there are two more agents, specifically designed for technical writing. The reporter agent takes an outline, and a folder of content, and it will write section by section, a detailed account of what the outline requested.

Unlike a standard LLM that might provide a high-level summary, the reporter is trained to expand specific placeholders with deep, evidence-based paragraphs. It draws directly from your research/ files and the project journal/ to ensure every sentence is grounded in the project’s actual state. Finally, the editor provides the final polish, auditing the draft for structural gaps and linguistic tics. It is grounded in a customizable style guide to make sure it always respects your style.

This distributed intelligence is held together by a central nervous system of context files, as per principle one. A journal/ directory provides a chronological record of decisions and progress, acting as a long-term memory for the project. The plans/ directory stores the strategic intent, while a TASKS.md file provides a high-level overview of the project’s current status. This structured environment allows the subagents to maintain a high degree of situational awareness without needing to ingest the entire repository in every turn.

Now that we have the key pieces in place, you can start to see why I think this approach is powerful. It is very extensible—you can add new agents triggered by specific commands to customize any kind of workflow—and it mostly solves the main pain point of modern LLMs, which is precisely the brittleness of long contexts.

Workflow Details

With all that, let me show you the specific commands and workflows I have currently implemented, but keep in mind what follows is but one example of the kind of powerful workflows we can start to automate.

I will divide the rest of the article into a few major areas, and explain the commands and agents that I use in each case, and a bit of the high-level instructions given to each of them.

Discovery & Strategy

The most critical phase of any project occurs before I write a single line of code. I call this the “Discovery and Strategy” phase, powered by the /research and /plan commands. By formalizing this process, I’ve moved away from impulsive execution toward a deliberate, architected approach.

The /research command is my primary tool for external knowledge. When triggered, the researcher scours the web for technical documentation and relevant case studies, synthesizing them into granular summaries in the research/ directory. These files become a persistent knowledge base, allowing me to reference verified facts without leaving my terminal.

Complementing this is the /plan command, which focuses on internal strategy. The planner conducts a thorough analysis of the codebase and the journal/ to understand the system’s current state. After an interactive dialogue to resolve any ambiguity, it produces a comprehensive Markdown plan in the plans/ directory. This document maps out the technical territory and provides a step-by-step execution roadmap.

The strength of this workflow lies in the synergy between these two tools. A /plan operation might reveal a gap in my understanding of a specific library, prompting a targeted /research session. Conversely, a new research finding might shift my technical direction, leading to a refined plan.

Software Development

Once I have a solid strategy in my plans/ directory, I can move into execution. Let’s focus now on software development. I’ve designed four core commands—/issues, /task, /commit, and /release—to eliminate the friction of context-switching between my IDE and my terminal.

The cycle starts with /issues and /task. The /issues command acts as an expert project lead, interfacing directly with the GitHub CLI to analyze open issues and recommend what to tackle next based on strategic impact. For roadmap tracking, the /task command manages a living TASKS.md document. It assesses the value of pending work to ensure my efforts are always aligned with the project’s goals.

As I translate the plan into code, the /commit command brings order to my workspace. Instead of a monolithic “wip” commit that hides the logic of my changes, the system analyzes the git diff and logically groups modifications into cohesive units. It separates a core feature update from a documentation tweak, then proposes a series of atomic, Conventional Commits for my approval. This keeps my version history pristine and easy to navigate.

The final stage is deployment. Manual releases are fragile processes fraught with repetitive checklists: bumping versions and running tests before managing tags. The /release command automates this entire sequence. It verifies the workspace integrity by ensuring a clean git tree and passing tests via make. It then analyzes the commit history to propose the next version bump, drafts a CHANGELOG.md entry, and publishes the final tag to GitHub. This transforms a tedious afternoon of housekeeping into a single-command operation.

But, as you’ve seen, everything happens in tandem with those principles. No important action is taken without my confirmation, and everything gets logged into the filesystem, so all future decisions are grounded in past experience.

Content Creation

Now let’s focus on writing high-quality documentation and long-form articles. This is perhaps the most sensible part of the article (and the system) because people are very sensitive today with the topic of AI writing—and rightly so. Again, my intention here is to enhance how I work and get stuff done. If you’re writing for the pleasure of doing it, that’s totally fine, you probably don’t want any help there.

Anyway, the approach is built on the same cognitive foundation as the development path: the research and plans gathered during the discovery phase should serve as grounding for writing.

It starts with the /draft command. In its initial phase, the system performs a deep scan of the research/ and plans/ directories to identify the key themes relevant to the requested topic. If the foundation is too thin, the system will pause and suggest a /research or /plan cycle to ensure the draft has sufficient substance. Once the context is validated, the workflow enters an interactive “Outline Creation” phase. Rather than guessing at a structure, the system proposes a detailed Markdown outline. This collaborative step allows me to set the narrative arc and logical flow that I want, iterating on the high-level structure of, say, a technical article, before committing on the details.

Once the outline is locked, the /draft process initializes a skeleton file—complete with section headers and strategic placeholders—and then moves into an iterative, section-by-section expansion. Here, the reporter subagent takes the lead. Guided by the specific context of each section, the reporter weaves together research summaries and technical specifications into professional prose, all grounded on a style guide document.

Because the expansion happens in granular steps, the system maintains a high level of detail that a single-shot generation would inevitably lose. The result is a first draft that is structurally sound and rich with technical depth.

However, a first draft is rarely the final word. It will always sound AI-ish, and for many other reasons, it is rarely good enough. To achieve professional quality, I use the /revise command, which runs a structural and linguistic audit powered by the editor subagent following the same style guide.

Unlike a simple “check my writing” prompt, the editor performs a deep analysis of the document’s flow and tone. It identifies logical gaps where more evidence might be needed and highlights awkward phrasing that could obscure my intent. And crucially, this isn’t an automated “fix-all” tool; it’s an interactive process. The system presents its findings and proposes specific improvements, which I can then review or approve.

This collaborative refinement process ensures the final output maintains a consistent, professional voice while benefiting from the speed of the AI. By using /revise, I can surgically improve the text to enhance clarity and impact without losing control over the narrative.

But, in any case, I always find necessary a manual review and editing after all the AI enhancements. It shouldn’t be a surprise to you that this article is written in this way, but what you’re reading now is probably 80% different to what the final /revise iteration gave me. There is only so much you can prompt an AI, and that final human touch is not part of it.

But that’s good. This automates the first 80% or so of compiling a gazillion sources into a coherent narrative, and leaves the remaining 80% of polishing for me, which is the part I actually enjoy about writing.

Background Tasks

But there’s more. All of the above is what happens during, let’s say, the work day. That’s me sitting in front of the terminal, typing commands, approving stuff, fixing and redirecting, etc. Being an orchestrator.

But the real magic of AI-assisted development is what happens when you’re not looking. How you can leave you AI assistant working through the night, compiling sources, fixing bugs and proposing pull requests, enhancing the test suite, burning tokens your behalf.

To achieve this, I built an automation layer via the /cron command. The heart of this automation is the cron.toml file. This configuration file allows me to define scheduled tasks with a simple, declarative syntax. Each task specifies a name, an execution schedule, and a natural language prompt for the AI to execute.

For instance, I can schedule a task to perform “Background Research” every midnight on the unfinished tasks, scouring the web for new developments in a specific technical niche or finding specific sources to deal with the recently discovered bugs. By offloading these repetitive tasks, I ensure the knowledge base remains fresh and the project’s momentum never stalls. When the morning arrives, we have a lot of new context to start planning the day’s bugfixes and feature developments.

Maintenance & Refactoring

Now, for the final touch, here’s how I deal with technical debt and feature rot. As a project evolves—and especially, as fast as AI-powered projects evolve—it accumulates technical debt—outdated implementations, untested paths, and plain old useless features—but also, contextual debt–—outdated plans and completed tasks that clutter the roadmap, and research we never acted upon.

Without deliberate intervention, this noise degrades the AI’s performance, leading to context rot. The /maintenance command is my primary defense against this entropy. It treats the development environment as a living instrument that I must regularly tune and sharpen to maintain its efficiency.

The /maintenance workflow follows the same plan-first architecture as the rest of the system. When invoked, the AI performs a comprehensive audit of the codebase, focusing on improvements like code readability and performance optimization. It identifies opportunities to apply the DRY (Don’t Repeat Yourself) principle and ensures that every function is documented with high-quality docstrings. But it also fixes deviations between the documentation and the actual implementation.

Crucially, this is an interactive process: the system presents a detailed refactoring plan for my approval before making any changes. This ensures that I remain in control while the machine handles the labor of cleaning the code.

Beyond code refactoring, I maintain system health through disciplined repository hygiene. A key component is the management of the TASKS.md file. By regularly moving completed items into the “Archive” section, I ensure that my primary operational view remains focused on what is relevant. This simple act of archiving prevents the “Active Tasks” list from becoming a source of distraction.

The goal of these maintenance practices is to provide the AI with the cleanest possible line of sight into the project’s state. When the repository is cluttered with stale research, the subagents are forced to sift through irrelevant data, increasing the risk of hallucinations. By treating maintenance as a first-class citizen, I ensure that every interaction, whether a /plan or a /draft, is grounded in a precise context.

Conclusion

This system is far from done, and as models improve in capabilities I’m sure we’ll unlock new areas for automation and augmentation that we cannot think about today. But for me, the key principles will remain valid for a long time. These are principles of robust engineering and management, after all. You can read them thinking of a completely human-based organization, and it’s all valid:

1. The important things should be made explicit.

2. Resist the urge to guess.

3. Delegate, delegate, delegate (yeah, three times).

And this is the key insight for me. Good AI users are basically good managers. All the science and engineering behind good practices for people management also apply to good AI management. And then there are of course technical considerations because AIs are not people, and perhaps never will.

So this is perhaps the most philosophical take-away from this article. Sorry to have made you read so long for this!

Now, on the technical side, please do check the repository and play with it. There are a couple extra goodies I haven’t tell you about, like a /scaffold command that creates new projects from scratch, and an /onboard command that explains the whole repository in detail. Those are particularly useful if you’re reusing this repository as a template to start your own project.

Ultimately, this repository is not a one-size-fits-all solution. It is a starting point. The commands and subagents provided here represent a particular opinion on how modern development should look, but they are not the only way. The power of this framework lies in its extensibility. Every system prompt for agents and commands is a living document, meant to be tweaked and rewritten to suit your unique mental model.

So if you do try it out, please let me know in the comments. And if you have a different (or similar) system set up for yourself, please share with all us your experience and your thoughts. We are all learners in this era of AI, and we can only help each other.

Stay curious.

This article is a quick recap of why I did it, how I went down the rabbit hole of implementing yet-another-drawing-library, and a small showcase of what it can do it can do, in case you want to give it a shot.

But before moving one, here is a minimal example of what you can do with a few lines of Python.

Are you curious now? Then read on!

Subscribe now

The Case for Tesserax

There are a gazillion tools out there for drawing mathematical diagrams in Python, from the ancient, trustable matplotlib to the new cool kid in the park manim. There are a ton of charting libraries (those intended to draw bars, pies, and doughnuts) like altair and plotly; and a ton of high-level diagram makers like graphviz for graphs and networks, and mermaid for, well, whatever the mermaid devs think is worth drawing.

The problem with all these is that, one, there weren’t made by me. Now, jokes apart, I do believe there is a lot of value in reinventing the wheel, if only because you get to learn a lot about wheel—and boy have I learned more than I ever wanted to about SVG in the last couple of weeks. But also, you may find that your wheel fits your cart slightly better than all others, just because you know precisely what you need (and enough with wheel analogies).

So, I made Tesserax to cover a sweet spot that I couldn’t find anywhere: a lightweight library (literally zero dependencies, not even numpy) that renders web-native content (so everything scales and layouts perfectly in Jupyter / Quarto) and has both a very powerful low-level engine for when you want pixel-perfect control, and a very comfortable high-level engine for the most typical workflows.

Also, it should support animations. But I’m getting ahead of myself.

What Can Tesserax Do

I won’t put any code in this article because, one, Substack sucks at code. And two, it would only make the article harder to follow. All of these examples are fully described in the online documentation.

At its core, Tesserax is a library for defining an SVG scene. You create a Canvas, and some Shapes to it, define some attributes, and render it as an SVG file. Simple enough1.

Beyond basic primitives, you can of course draw arbitrary paths and style them easily. Here is a Polyline primitive that allows controlling the curvature with a smoothness factor.

You can control and distort these shapes in any form you want, including fully procedural warping of the edges.

And, of course, since no mathematical drawing library is worth a penny without a sketchy mode, you can also do that (albeit with some constraints).

To make positioning and layout easy, Tesserax comes a set of builtin layouts (that of course can be extended in any way you want, it’s Python) for common patterns like rows, columns, grids…

And a couple of complex layouts for things like trees and arbitrary graphs.

But things start to get really interesting when you discover that if you can render an image from code, then you can render as many as you want! And that leads to… animations!

Tesserax animations are procedural, meaning you write code that defines how each object changes in time. But there is a very high-level API for defining and composing animations declaratively, which means you seldom need the full power of tweaking each shape properties (but that power is there when you need it, as it should).

And, of course, once you grok animations, the next immediate idea is physically-based animations. (This is where things got way out of control, as you may imagine). Tesserax comes with a very small but fully-fledged pure Python 2D physics engines—that was a mouthful.

It can resolve collisions among arbitrary objects (using circles and rotated boxes as approximate colliders) and simulate fixed-step rigid body mechanics. Physic animations are baked—which means you compute the animation once and then render it. So, no, no 2D games for you! What were you thinking about? This is a graphing library, for Turing’s sake!

The purpose of this physics engine is, once again, didactic—e.g., to explain physics concepts in a Jupyter notebook with a bit of shizzass.

Final Words

I started writing Tesserax to help me create diagrams and animations for my lectures and articles. But when you do that with a fully-fledged programming language things start to get really interesting, because now you can leverage your well-honed programming skills to create complex diagrams with the minimum necessary code. You can encapsulate repeatable patterns into classes and methods, and build impressive diagrams bottom-up. You can abstract common patterns into drawable concepts like trees, automatons, etc.

For example, in the docs you will find a bit of code to implement a blob-like simulation by creating a ConvexHull component that automatically tracks inner shapes and builds the convex surrounding path around them. Tied with some simple physics (a bunch of balls interconnected with springs) gives you this ugly-looking blob of dread that I hope haunts your dreams for the next week or so—it sure will mine.

On a more serious note, the next step for Tesserax is building a library of reusable math and computer science concepts, like animated arrays, trees, graphs, etc., that can be used to explain complex CS topics with the much need visual assistance. As the great Donald Knuth once said, an algorithm must be seen to be believed. Tesserax is here to help you see them.

I’d really love if you guys would give Tesserax a try, now that is a bit more polished, and shoot me all your questions and suggestions. Special thanks to for many insightful comments and suggestions (although I’m not sure I’ve totally addressed all of them :)

The following is the first draft of Chapter 1 of How to Train your Chatbot, a developer-centric book I’m writing in public. Subscribe to get all chapters in your inbox, as they come out, for free.

Subscribe now

In this first chapter we will build a bare-bones conversational application–a chatbot–that is capable of maintaining a coherent conversation with a single user. We will learn how to connect with an LLM provider, pass messages, and stream the responses back.

In this chapter we won’t use any additional libraries beyond the Python OpenAI wrapper to communicate with an LLM backend. This is on purpose, so we understand the whole lifecycle of a typical conversation.

Then, in the next chapter, we will redo the core logic using ARGO to simplify the chat management and introduce some modularization, and we will add a database to store conversations and make our interactions persistent.

The whole source for this chapter (and this entire book) can be found in this Github repository. This specific chapter is in the basic.py file.

Setting up the environment

The first step is to setup a development environment. At the moment, the community preferred way of doing this with Python is using a package and virtual environment manager, and by far the best in the world is uv. So let’s begin by installing thatl, and once ready, just run:

mkdir chatbot # or whichever folder name you prefer
cd chatbot
uv init .

This will create a pyproject.toml file in the chatbot folder. Now let’s add streamlit and openai.

uv add streamlit openai

After a few seconds downloading the necessary packages, you will be ready to start coding.

Get the source code

The complete source code for this book is available in Github with an MIT license. Feel free to fork it and use it as you see fit.

The basic chatbot

Let’s start outlining our first chatbot. We will use Streamlit’s builtin functionality to create a pretty decent chatbot UI in less than 10 lines of code. Here is the boilerplate code.

import streamlit as st

# import and setup OpenAI
<<setup_openai>>

# regenerate the conversation history
<<render_history>>

msg = st.chat_input()

if not msg:
    st.stop()

with st.chat_message(”user”):
    st.write(msg)

# code to actually call the LLM
<<invoke_llm>>

with st.chat_message(”assistant”):
    response = st.write(reply(msg))

# save the llm response
<<save_llm_response>>

About Literate Programming

You may have noticed the code above uses a somewhat weird notation, like <<setup_openai>>, which is not valid Python. This is because we are using a tool called illiterate to generate automatically the source code from the book content. This notation is how we reference snippets of code that will be defined later on.

This is a paradigm called Literate Programming that emphasizes narrative documentatiopn as the primary source, and the actual source is derived automatically, so that your documentation (this book) and your code are always synchronized

Don’t worry too much about it, just know that all these snippets will be explained in the remainder of this chapter. You’ll get the hang of it.

This completes the basic layout of a typical chatbot app. We still haven’t specified how to actually call the LLM provider, and how to manage the conversation history. We will do that in the remaining of this chapter.

Setting up an LLM provider

For all practical purposes, when you code a chatbot you consider the LLM as a black-box component that is, almost always, encapsulated behind a REST API. This lets you focus on building the app and forget about the extremely complex problem of serving, scaling, and monitoring an LLM in a production environment.

This is pretty much the same as when you use a production database, like MySQL, Postgre, or MongoDB, you almost never directly run the database server daemon. Instead, you almost certainly provision a managed server from Amazon, Azure, Google, or any other of the myriad cloud providers. In the same way, most of the time you won’t run an LLM locally, but instead use a cloud provider which has far better infrastructure.

By far, the most common API design used by all cloud LLM providers is the OpenAI API, which has become a de-facto standard. So, even if you don’t use OpenAI’s models directly (like GPT-5), you will almost certainly use the opena Python package to communicate with your LLM cloud provider, whoever they are.

In this book I will recommend OpenRouter, because they provide access to a huge range of models, and they deal with automatically routing to the optimal cloud provider. But you are free to choose whatever LLM provider you prefer, and as long as they give you an OpenAI-compatible API (and they all do), the rest of this book will work exactly the same for you.

Whatever you choose, you will need three things from your LLM provider:

• A base URL (e.g., https://openrouter.ai/api/v1)

• An API key (e.g., sk-12345678...)

• A model identifier (e.g., meta-llama/llama-4-maverick:free)

Of these three, the API Key is the most important to keep safe (and secret) because it’s what the LLM provider will use to identify you (and thus charge you) for using their service. If this API Key ends up in a Github repository or somewhere public, someone might use it to impersonate you and thus deplete your credits or put you in a significant debt with your LLM provider.

Once you have identified your LLM provider, you need to provide streamlit with the credentials and metadata indicated above. The best way to do this for a streamlit app is to use its native secrets management. You need to create a .streamlit folder, and place a secrets.toml file inside. There you can add the entries just as if they where environment variables.

# .streamlit/secrets.toml
api_key=”sk-12345678”
base_url=”https://openrouter.ai/api/v1”
model=”meta-llama/llama-4-maverick:free”

Needless to say, this secrets.toml file should be the first line in your .gitignore!

For the purpose of development and testing, you can use one of the many free models OpenRouter hosts, which give you enough daily uses for development purposes. Read more on the OpenRouter documentation. As a perhaps obvious disclaimer, I’m not associated with OpenRouter nor any part of this book is supported by them. I just think they provide a wonderful service for the AI developer community.

Once you’ve gotten your API key, you can use it to create an OpenAI client to interact with the LLM.

from openai import OpenAI

client = OpenAI(
    base_url=st.secrets.base_url,
    api_key=st.secrets.api_key,
)

And we’re ready to start sending some messages.

Calling the LLM

OpenAI’s client makes it very easy to invoke an LLM. We just need to pass in a list of messages and stream the result back to our Streamlit app. But before, we need to understand one key thing about LLM providers, and is the fact that they are stateless. This means you always have to pass the whole conversation to the API, because neither the actual model nor the server itself “remember” the conversation.

So, we will need to store the conversation in our application state. I’ll show you how to do this in just a second, but let’s assume we have it and move on to actually talking to the LLM.

In our app, we need to include the following method, which will stream the response from the LLM to the user.

<<store_conversation>>

def reply(msg: str):
    # retrieve the conversation list
    conversation = st.session_state.conversation

    for chunk in client.chat.completions.create(
        model=st.secrets.model,
        messages=conversation,
        stream=True,
    ):
        if msg := chunk.choices[0].delta.content:
            yield msg

That’s it. We just invoke client.chat.completions.create with the model name and conversation (a list of messages we will see right away) and stream the results back in the form of a Python generator. The st.write method in our main app is smart enough to produce a pretty typewriter animation as we receive the response one chunk at a time.

The conversation is actually stored in Streamlit’s builtin session storage, which is a dictionary-like object unique to each user session. This means that as long as we stay in the same browser tab, the conversation persists. Once we open another tab or hit F5, the session storage is cleaned up and we have a new conversation. Kind of brute, but for now it will do.

And this is the missing snippet to actually keep track of the conversation. We need to store the user message just after we get from the chat input widget:

if “conversation” not in st.session_state:
    st.session_state.conversation = []

st.session_state.conversation.append(dict(role=”user”, content=msg))

And we need to store the assistant reply message in the main loop after streaming it. Fortunately, the st.write method is smart enough so that it not only prints the message as it is streamed, but also returns the full concatenated message once the stream is closed.

st.session_state.conversation.append(
    dict(role=”assistant”, content=response)
)

As you’ve seen, we’re storing the messages in the format that the OpenAI API expects. This is a list of dictionaries, where each dictionary contains a role and a content key. The role (for now) can be either user or assistant, and it’s important we respect it because the models are trained to behave as if they are replying from the “assistant” perspective.

Rebuilding the conversation history

The final step to make our app a fully-fledged chatbot is to actually render the whole conversation in the UI. This is necessary because Streamlit works with an immediate execution paradigm, which effectively means every interaction cleans the whole screen and redraws the entire UI. Thus, every time the user enters a message, the UI restarts and we lose the previously rendered interaction.

Fortunately, as we have the conversation stored in the session state, we can simply iterate over it and render each message.

for msg in st.session_state.get(’conversation’, []):
    with st.chat_message(msg[’role’]):
        st.write(msg[’content’])

We just need a bit of care the first time we run the app because the conversation key won’t exist, that’s why we use get instead of a direct indexer.

Closing up

Phew! There you go, in around 50 lines of code (including comments and whitespace) we have a fully working chatbot. It’s very crude for now, all it does is send some messages to an LLM provider and stream the response back, but this is the foundation we can build upon to create all sorts of cool stuff.

In the next chapter we will rebuild this application from scratch but using an LLM framework that will give us much more flexibility later on.

Subscribe now

TL;DR: In this upcoming series, we'll build an autonomous LLM-based agent from scratch, focusing on the fundamentals. We'll go from zero to fully autonomous deep research, one feature at a time, staying as close to the metal as possible. Subscribe to receive all posts in your inbox for free.

Subscribe now

Have you looked at the latest AI breakthroughs—agents that can plan, code, and research—and thought, "That's cool and all but… how the hell do one actually build something like that?"

If you're just getting started with Python development, or even if you’re a seasoned expert, you've probably felt this gap too. Maybe you've played with the demos, called an API or two, and seen just enough of the magic to get excited.

But moving from just prompting an LLM to building intelligent, autonomous agents is a different beast entirely. Things start to break pretty soon as you stack up more and more complexity. And it seems every tutorial out there is oblivious to this. Anyone can show you how to code a simple chatbot maybe with some RAG, but very few resources take you from that humble beginnings to something that actually looks like a modern multi-agent system that works autonomously.

This series is my answer to that problem.

Over the next 10 or so posts—the next couple of months—we are going to build a comprehensive, autonomous AI agent. And we're going to do it incrementally.

We'll start with the simplest possible application: the basic chatbot loop that stores conversation history. Why? Well, because even if everyone has already seen this, we need proper foundations to build on top. We need to see the full flow—how messages are stored, how a system prompt works, how to interact with the LLM, and what's really happening under the hood.

But from there we'll level up pretty fast, adding new capabilities piece by piece until we have a personal agent capable of doing deep research and generate long, coherent reports all on its own.

Tools of the Trade

Now, you might be thinking, why build from scratch and not just use LangChain, or LlamaIndex, or some other well known framework?

Here's my thesis. When you start with those massive, professional frameworks, you learn more about the framework than you do about the fundamentals. Those tools are powerful, but they are also black boxes. They hide the logic and complexity behind layers of abstraction. You end up learning their way of doing things, their design patterns, and their API. And frameworks change, but fundamentals remain.

I believe it's far more valuable to learn how to do things as close to the metal as possible. Our goal is to write the least amount of boilerplate code possible, but—and this is the key—without hiding the any of core logic. We will build this entire thing ourselves, understanding every single piece of the puzzle.

We'll use a tech stack that gets out of our way. Only three tools:

For the UI we’ll use Streamlit. For those who don’t know it, Streamlit is a radically simple application framework that turns any Python script into a fully fledged web app. The key here is what it doesn't make you do: you write zero boilerplate, zero layout, zero presentation, and zero state management code. You just write core logic, and a good enough web UI appears.

For the actual chatbot we'll use the ARGO framework. I like to call it FastAPI for AI agents because it has that same lightweight, decorator-based feel. It is the simplest, most Pythonic agent framework you’ve seen, and I promise you’ll fall in love with it. Despite being super simple, it forces you to build the logic using a very clean and lightweight skills-based pattern for modularizing our agent's intelligence.

And for data we’ll BeaverDB. This is a lightweight wrapper on top of SQLite that gives you everything you could ever want from a modern database. But it is not a modern database server. It's just a single, embedded binary file on your disk. There's no Docker, no authentication, no connection strings, no schemas, no boilerplate. Yet, on top of this simplicity, BeaverDB gives us a very Pythonic, very comfortable API for a document database with vector and full text search, and cool features like persistent dictionaries, lists, priority queues, and even pub/sub. It makes building something like a RAG pipeline extremely simple without any complex setup.

That's it. With just these three basic libraries, we will build a fully autonomous agent.

The Journey Ahead

Here's the roadmap I've thought so far. We'll add a new level of intelligence in each post.

• Level 0: The Conversationalist. The foundation. A simple Streamlit UI with a stateful chat loop that understands conversation history and system prompts.

• Level 1: The Assistant. We'll give our agent long-term memory. We'll build a full Retrieval-Augmented Generation (RAG) pipeline from scratch using BeaverDB so it can answer questions based on a private knowledge base.

• Level 2: The Researcher. We'll break our agent out of its box and give it a tool to search the web, allowing it to access up-to-the-minute information.

• Level 3: The Analyst. This is where it gets really fun. We'll give the agent the ability to write and execute its own Python code in a safe sandbox to solve problems, analyze data, and even debug its own mistakes.

• Level 4: The Editor. We'll upgrade our UI from a simple chat to a persistent, shared canvas, where you and the agent can work side-by-side to co-write documents and refine plans.

• Level 5: The Scientist. The finale. We'll integrate all previous capabilities, giving the agent a master prompt that allows it to take a high-level goal, create a multi-step plan, and execute it independently from start to finish.

So here is my promise. It may take us more than 5 posts, but we are going to learn how to build a chatbot by building a chatbot, step by step. No unnecessary theory, just working code and clear intuitions.

I'll be providing a public GitHub repository with all the code, and each post will be a direct, straightforward Python script that you can type in half an hour.

When you're done with this series, it won't matter if you want to keep using ARGO and BeaverDB—which I hope you will—or move on to more “professional” tools. You'll be ready anyway, because these tools are so thin, what you'll have learned isn't how to use a specific library. You will have learned how the logic works underneath.

You'll understand how to orchestrate the different components involved in a typical chatbot, how to modularize the agent's brain, and more generally, how to think about building capable, intelligent, LLM-based applications. That's the real takeaway.

In the next post, we'll jump straight into Level 0: The Conversationalist. So hit that subscribe button and you'll get all posts in your inbox, 100% free.

Subscribe now

Ready to build?

I really love to code. That much I’m sure you can tell. So I have this very common problem, that I’m sure many of you have. A gazillion ideas in my head wanting to burst out, and very little time to try them all. To try and get around this, I like to prototype really fast and dirty implementations of anything that makes me curious, just to see if it leads somewhere. (This is why I have hundreds of unfinished projects in my Github.)

So, where was I? Yeah, this is one of the reasons I’ve made Python, and specifically streamlit, a key part of my toolkit. It lets me build the very simplest semblance of an app extremely fast, without considering anything remotely unnecessary (for a first prototype) like authentication and user roles, or worrying about the user interface.

Now, here is the thing. Recently, I’ve been building a lot of AI stuff. Chatbots galore for all sorts of tasks from general question answering to specific things like storytelling, evolutionary computation, automated coding, you name it. And I keep finding a specific pain point that grinds me to a halt in the most exciting part of prototyping: the database.

Very soon, my simple AI prototype needs a vector database for RAG or a key-value store for simple configurations, or a message queue for background tasks and a persistent list to store conversation history. And this is before anything remotely close to a production-ready feature is in the horizon. This is just the second thing you’ll need right after streaming the first few tokens from OpenAI. It’s unescapable for anything but the simplest toys.

So, before I know it, I’m wrestling with two or three containers—or worse, juggling credentials for three different cloud services—and writing boilerplate code just to glue it all together. I’ve become an accidental, and very grumpy, DevOps engineer. My worst nightmare!

What I want is something in the spirit of streamlit but for managing data. A minimalistic, no-bullshit database that just supports the basic modalities we all need in modern applications, like, dunno, JSON storage, maybe? And vector search? Yeah, that, but also combined with full-text and fuzzy, persistent lists and queues… hell, even a decent pub-sub system if we’re asking! Is it too much to ask for someone to make SQLite but for modern data!

Well… I guess if you want something done, you might as well do it yourself.

So this is the story of how I built BeaverDB—a Pythonic and multi-modal interface on top of SQLite that just works out of the box for rapid prototyping. And it scales to medium sized projects just fine (SQLite is damn fast!). BeaverDB my attempt to build the tool I wish I had—a library that provides the high-level data structures modern AI applications need, without sacrificing the radical simplicity of a single .db file.

Introducing Beaver

The guiding principle behind Beaver’s API is a minimal surface area with a fluent interface. You only ever need to use two classes. Everything else flows naturally from the main database instance, returning dedicated wrapper objects with a rich, Pythonic interface.

One key idea for this design is that it should just work. Zero configuration, just sensible out of the box decisions. So, no schema, everything indexed by default, no need to declare tables or entities or anything before hand, collections just get created when first used, you know, that kind of thing. It should be as easy as instantiating a class and calling two methods.

Let’s take a quick tour on what you can today with it.

Key-Value & Caching

Need to store configuration, user profiles, or cache expensive API calls? The namespaced dictionary is your go-to. It behaves just like a Python dict but is backed by the database, with optional TTL support.

# Use a dictionary for caching API calls for 1 hour
api_cache = db.dict("api_cache")
api_cache.set("weather", {"temp": "15C"}, ttl_seconds=3600)
print(f"Cached weather: {api_cache.get('weather')}")

Persistent Lists

This is the most straightforward way to manage ordered sequences. For a chatbot, it’s the perfect way to maintain the turn-by-turn history of a conversation. It works like a Redis list, with all bells and whistles, implementing a full Pythonic list interface, but backed in the DB. It supports appending and removing from the head and tail, as well as inserting or removing anywhere in between, blazingly fast, because all operations are indexed.

# Manage a conversation with a user
chat_history = db.list("conversation_with_user_123")
chat_history.push({"role": "user", "content": "Hello, Beaver!"})
chat_history.push({"role": "assistant", "content": "Hello! How can I help?"})
print(f"Conversation length: {len(chat_history)}")

Priority Queues

A priority queue is the essential tool for orchestrating an autonomous agent. It ensures the agent always works on the most critical task first, regardless of when it was added. The API is extremely simplified on purpose. For anything more complicated, use a full-featured list.

# An AI agent's task list
tasks = db.queue("agent_tasks")
tasks.put({"action": "summarize_news"}, priority=10)
tasks.put({"action": "respond_to_user"}, priority=1) # Higher priority

# Agent always gets the most important task first
next_task = tasks.get() # -> Returns the "respond_to_user" task

Real-time Pub/Sub

Need to build a decoupled, event-driven system? The pub/sub channel allows different parts of your application—or even different processes—to communicate in real-time. Beautifully designed with an extremely simple fluent API, but extremely performant, thread-safe, even works across different processes. Plus, it comes with an optional async interface if you’re feeling fancy.

# In one process, a monitor publishes an event
system_events = db.channel("system_events")
system_events.publish({"event": "user_login", "user": "alice"})

# In another, a logger subscribes and receives the message
with db.channel("system_events").subscribe() as listener:
    for message in listener.listen(timeout=1):
        print(f"Event received: {message}")

Collections & Hybrid Search

This is the core component for any Retrieval-Augmented Generation task. It’s a multi-modal collection of structured documents that understands vectors, and text, allowing you to combine search strategies for the best results. It also performs fuzzy search on demand with a very clever indexing strategy that I’ll tell you all about in the next section.

from beaver import BeaverDB, Document
from beaver.collections import rerank

docs = db.collection("articles")
doc = Document(embedding=[...], content="Python is fast.")
docs.index(doc, fuzzy=True)

# Combine vector and full-text/fuzzy search for better results
vector_results = docs.search(vector=[0.15, 0.85, 0.15])
text_results = docs.match(query="pthon", fuzziness=1)
best_context = rerank(vector_results, text_results)

And there’s a lot more. You can connect documents with relations and build a knowledge graph that you can later query to find similar documents or implement graph-based recommender system.

All in one freaking database file. No docker. No servers. No headache.

A Peek Under the Hood

Beaver is built on a series of pragmatic design decisions intended to leverage SQLite’s native capabilities to the fullest, avoiding slow application-layer logic wherever possible.

For one, it never creates new tables when storing stuff. Everything is stored and indexed in cleverly designed global tables that are created at startup time (only the very first time the DB file is created). This also means you get virtually infinite lists, dicts, queues, collections, etc., because these aren’t different tables (which would be a pain in the arsenal to maintain). And… (roll drums)… no migrations!

I want to highlight two specific features to tell you a bit about the underlying implementation details, so you can see the lengths it goes to try and be efficient out of the box.

The Pub/Sub System

The pub/sub system is the greatest example of efficiency by design. It’s built on a single, append-only log table with an index for the channel name.

For each channel, a single background thread polls this table for new entries and fans them out to any number of in-memory queues, one for each subscriber. The key insight here is that because the database is only ever touched by one polling thread per channel, adding a second, third, or hundredth subscriber adds zero additional load to the database. From the database’s perspective, new listeners are basically free.

Even better, this polling is only activated when there is at least one listener, and stops immediately after the last listener disconnects. This means we only ever use the minimum resources necessary.

Hybrid Text Search: The Two-Stage Filter

The other feature I think is kinda beautiful is the fuzzy search. Calculating the Levenshtein distance between two strings is computationally expensive, and there is no way to build an index beforehand without a combinatorial explosion in storage size. But running it across every document in a collection would be unusable for anything beyond a few thousand entries.

Beaver solves this with a two-stage process. First, it uses a pre-computed trigram index in SQLite to instantly find a very small set of candidate documents that share a high percentage of 3-character “chunks” with the query. This is a very fast SQL query that already gives us a very good approximation to the response. Then, we run the expensive Levenshtein algorithm only on this small, pre-filtered set of candidates in memory.

The Vector Store Dilemma

Vector storage is my biggest concern at the moment. The current implementation is, I think, pretty good for a typical use case of infrequent indexing and fast retrieval. But it can be much improved.

Right now, Beaver uses an in-memory k-d tree, which provides excellent search speed once the index is loaded into RAM. However, the index is ephemeral—it lives and dies with the application process. This creates a significant bottleneck: every time your application starts, the entire index must be rebuilt from scratch by reading all vectors from the database. Furthermore, indexing a new document requires a full, blocking rebuild of the entire tree.

As long as you index documents in a background process, infrequently—which is what most RAG-based applications do—this works just fine. But I’m not satisfied with this implementation, so here is my plan for its update.

The roadmap goes through integrating a state-of-the-art ANN library like faiss. This will enable a persistent, on-disk index that can be loaded instantly. But more importantly, we need to support incremental additions, which means newly added documents don’t rebuild the entire index. The key to achieve this is to have a persistent, large index, and a small, in-memory, temporal index of new additions that gets added to the persistent index from time to time. Keep these two in sync is far from trivial, but I already have a somewhat detailed plan.

Conclusion

Beaver is the result of a couple of weekends of focused frustration, and it’s already the backbone of my own AI prototypes. Even if it never reaches any level of maturity, I’m extremely satisfied with it because it already works for my use cases, and I’ve learned so much building it. Damn, who knew building databases could be so fun!

Now, what’s in it for you? Well, Beaver is not meant to replace your production PostgreSQL cluster. That’s unthinkable. But it might be the “good enough” database that lets you go from an idea to a working prototype in minutes, not hours. So, stop being an accidental DevOps engineer and go build something cool.

You can get started with a simple pip install beaver-db. Check out the GitHub repository for a lot of examples. And leave me any questions or comments either here or in the repository issues.

Have fun!

Hey! Alejandro here, from Mostly Harmless Ideas. Welcome to Coding Lessons, a section of MHI where I sparingly post short, coding-focused articles. This one came out after a couple of days struggling to find the right balance for managing complexity in a recent project. Hope you enjoy it.

If you’ve ever worked on any application with a development and a production environment, you’ve dealt with this problem: managing configurations and feature flags. A centralized, easy-to-modify location for configurable options that vary across different deployment environments is crucial for maintaining flexibility and enabling rapid iterations. I’ve been recently working on a mid-sized chatbot project that requires a way to toggle specific features in different environments (think enabling different kinds of tools or skills for the chatbot).

After struggling with environment variables and looking over some really over-engineered solutions, I found a straightforward, pure Python approach that works perfectly for small teams that want to iterate fast. This article is a short presentation of this approach. We will explore the motivation behind this method, provide implementation details, and highlight potential pitfalls to avoid.

Motivation

As applications grow, so do their configurations. Development, staging, and production environments often require distinct settings. If not appropriately managed, managing these configurations can become cumbersome. Traditional approaches involve plain text files like a .env or config.toml file—that are not expressive enough—, using structured formats like a config.json or config.yaml file—which are more expressive but harder to read and maintain—or worse, relying on external libraries to manage configurations, leading to complexity and confusion.

The proposed method simplifies this by utilizing separate Python files for each environment. By dynamically loading the appropriate configuration based on a single environment variable, we can keep our code clean, organized, and easily managed.

An example of a well-established framework that follows this practice is Django. Django projects often separate environment-specific configurations into distinct Python files, such as development.py and production.py, to manage settings for different environments effectively. This approach is widely recommended in the Django community as it simplifies transitions between environments and allows for better organization of configuration settings.

Implementation Details

The core idea of this method is to put configuration options and feature flags in a Python file so that, once imported, you can access all those variables as module members. The only real trick is to dynamically import the correct file based on an environment variable, so you get development, staging, testing, or production configuration (or whatever contrived scenario you have) without the rest of your code knowing what environment is running on.

Here’s a plain step-by-step process.

Step 1: Create Environment-Specific Configuration Files

We will create separate configuration files for each environment (e.g., production.config.py, staging.config.py, development.config.py). Each file will define global variables for feature flags and other configuration options.

Example: production.config.py

NEW_FEATURE = False 
EXPERIMENTAL_FEATURE = False 
DATABASE_URL = "https://prod.database.url"`

Example: staging.config.py

NEW_FEATURE = True 
EXPERIMENTAL_FEATURE = False 
DATABASE_URL = "https://staging.database.url"`

Example: development.config.py

NEW_FEATURE = True 
EXPERIMENTAL_FEATURE = True 
DATABASE_URL = "https://dev.database.url"`

Step 2: Create a Loader Function

Next, we will create a config.py file that contains a function to load the appropriate configuration based on the ENVIRONMENT variable.

import os
import importlib

def load_config():
    environment = os.getenv("ENVIRONMENT", "DEVELOPMENT").upper()

    try:
        return importlib.import_module(f"{environment.lower()}.config")
    except ImportError:
        raise Exception(f"Configuration for environment '{environment}' not found.")

Step 3: Access Configuration in Your Application

Finally, we can access the loaded configuration in our application by calling the load_config function.

from config import load_config

def main():
    config = load_config()

    if config.NEW_FEATURE:
        print("New Feature is enabled!")
    else:
        print("New Feature is disabled.")

    print(f"Database URL: {config.DATABASE_URL}")

if __name__ == "__main__":
    main()

Step 4: Set the Environment Variable

Before running your application, set the ENVIRONMENT variable in your terminal (or in a .env file if you are already using it):

export ENVIRONMENT=production python main.py
# or staging or development

Advantages and Limitations

When managing application configurations and feature flags, developers often face a choice between using environment-specific Python configuration files and traditional .env files that are most commonplace. Each approach has its own set of advantages and limitations. Here, we will explore these aspects in detail.

Advantages of Python Config Files

1. Version Control: Python configuration files can be checked into version control systems (like Git) without exposing sensitive information. This allows teams to track changes, collaborate effectively, and maintain a history of configuration adjustments across different environments.

2. Expressivity: Python is far more expressive than any plain-text or structured format. You can have options that are lists, dictionaries, or even instances of custom classes. You shouldn’t abuse this power, though.

3. Computed Values: Python files allow for the inclusion of computed values and logic. This means you can dynamically generate configurations based on other parameters or conditions, providing greater flexibility than static key-value pairs in .env files.

4. Modular Design: Common configuration values can be refactored into separate modules imported by relevant configuration files. This modularity promotes code reuse and reduces redundancy, making maintaining configurations across multiple environments easier.

Limitations of Python Config Files

1. Security Concerns: Unlike .env files, which are often excluded from version control (ensuring sensitive information remains private), Python configuration files may inadvertently expose sensitive data if not managed properly. Developers must exercise caution to avoid committing sensitive information to the repository.

2. Risk of Misconfiguration: The flexibility of Python configuration files can lead to developers unintentionally modifying production settings during development or testing phases, potentially causing disruptions in live environments.

3. Learning Curve: This approach may not be as familiar to new developers as the widely adopted .env file convention. As a result, onboarding new team members may require additional training or documentation to understand the custom setup.

Advantages of .env Files

• Simplicity: .env files are straightforward text files that store key-value pairs, making them easy to read and edit.

• Common Practice: The use of .env files is a well-established practice in many development communities, making it easier for new developers to adapt.

Limitations of .env Files

• No Security by Default: While .env files can be excluded from version control, they are still plain text files that can be easily accessed if not adequately secured.

• Limited Functionality: .env files do not support computed values or complex logic, which can limit their usability in more dynamic applications. However, this may be considered a strength in reducing configuration complexity.

Striking a Sane Balance

There is no free lunch, or so the saying goes. However, you may strike a sane balance in this case by externalizing sensitive and static configuration options to .env files and keeping more dynamic options (e.g., feature flags, option lists, or computable values) in Python files. This way, you maintain flexibility for rapidly changing or very development-specific configuration variables and leave the most sensitive, project-wide configurations for maintainers or DevOps engineers to handle.

Obvious Pitfalls to Avoid

While this approach offers a clean and organized way to manage configurations, there are some pitfalls to be aware of:

1. Security Risks: I already said it, but to reiterate, DO NOT STORE sensitive information (e.g., API keys and database credentials) in plain text within your configuration files. Instead, consider using environment variables or, even better, secret management tools.

2. Version Control: As a follow-up to the above, ensure that sensitive information (e.g., what’s included in .env files) is excluded from version control (e.g., using .gitignore). This helps prevent accidental exposure of secrets.

3. Error Handling: Implement robust error handling when loading configurations. If a specified configuration file does not exist or fails to import, your application should gracefully handle this in upstream code rather than crash unexpectedly.

4. Documentation: Clearly document each configuration file and its purpose. You have Python comments, for Turing’s sake! This will help team members understand the differences between environments and make necessary modifications.

5. Testing: Regularly test your application in all environments to ensure that the configurations are correctly applied and that features behave as expected.

Conclusion

Managing feature flags and configurations in Python can be streamlined using environment-specific configuration files loaded dynamically based on a ENVIRONMENT variable. This approach enhances organization, simplifies edits, and provides clarity when dealing with multiple environments. By being mindful of security risks and pitfalls, you can implement this strategy effectively in your projects, leading to smoother development cycles and more robust applications.

Choosing between Python configuration files and .env files depends on your project's needs and the team's familiarity with each approach. While Python configuration files offer greater flexibility and modularity, they come with risks that require careful management. On the other hand, .env files provide simplicity and are widely recognized but lack built-in security features and advanced capabilities.

Ultimately, understanding the advantages and limitations of each method will help you make an informed decision that aligns with your development practices and security requirements.

Now, I’m interested in reading your thoughts. Have you ever used Python configuration files? If not, why not? If yes, what was your experience?

Note: This article is part of my in-progress ebook, How to Train your Chatbot, a jam-packed handbook on how to use LLMs to build all sorts of cool stuff. You can get early access to the ebook (PDF and EPUB) as well as the full source code of all demos in the following link:

How to Train your Chatbot - 50% off

In this article, we will continue our exploration of retrieval augmented generation, but zooming out to the widest possible scale. Instead of answering questions from a restricted domain, like a PDF, let’s tackle the opposite problem: a bot that can answer anything at all!

The problem is our bot doesn’t know about everything that has happened in the world, and especially it doesn’t know about anything that has happened since its base pretraining.

But you know who does? Google! Well, not exactly everything, but a giant search engine like Google is the closest we have to a repository of the world’s knowledge. However, as everyone who’s ever used a traditional search engine knows, it is often not trivial to find the answer to your question. You have to sift through several of the search results and skim through their content.

On the other hand, LLMs, even the most powerful and recent ones, are by definition static, in the sense that you cannot update their knowledge without some costly retraining. And even if you could, there is still a limit to how many raw facts they could learn and then retrieve flawlessly. As we’ve seen, LLMs are lousy at remembering minute details like dates and are prone to subtle hallucinations.

So, how can we combine the language understanding and synthesis power of LLMs with the recency and breadth of knowledge of search engines?

With RAG, of course!

Note: If this is the first article of this series for you, make sure to check the previous articles here and here, and my guide to retrieval augmented generation. This article bases heavily on previously discussed concepts.

The plan

Keeping in line with the previous article, I will first explain at a broad level what is the workflow and the necessary functionalities for this app, and then we’ll go into implementation details. Since much of this demo builds on previous functionality, I won’t show you the whole code, only the novel parts.

You can check a working version of this demo online at https://llm-book-search.streamlit.app. This demo is provided on a best-effort basis and might be taken offline at any moment.

At a high level, this demo looks like a poor man’s search engine. We’ll have a simple search bar where the user can type a query. But actually, this isn’t just a search engine, it’s an answer engine! Instead of listing the matching websites, we’ll take it a step further and crawl those websites, read them, find a relevant snippet that answers the user query, and compute a natural language response with citations and all. Furthermore, we won’t even pass the exact user query to Google, but instead ask the LLM to suggest a small set of relevant queries that are more suitable for search.

Here is a screenshot of the final result, showing our answering engine talking about recent news (the launch of GPT-4o) which happened way after its training:

Here is the general workflow:

1. The user types a question.

2. The LLM provides a set of relevant related queries.

3. For each query, we obtain a set of search results from Google.

4. For each search result, we crawl the webpage (if possible) and store the raw text.

5. Given the original user question and the crawled text, we build a suitable context.

6. The LLM answers the question with the provided context.

There are several problems we must solve to implement this workflow, and we will go over them one at a time.

Getting the relevant queries

The first step is to compute a set of relevant queries related to the user question. Why? Well, first, because the user question might not be the type of query that search engines are good at solving. If you’re using Google, this might be less of an issue. But if you apply this strategy to self-hosted traditional search engines, like ElasticSearch, they instead just use some fancy keyword-based retrieval, with no semantic analysis.

But more importantly, there simply might not be a single best document (or webpage) with an answer to the user question. For example, if the user asks about a comparison between two products, but there is no actual comparison in the data. All you have is separate documents describing each product. In this case, the best strategy is to perform separate queries for each product and let the LLM do the comparison, effectively leveraging the best of both paradigms.

Here is a prompt to achieve this. It’s a simple but effective combination of prompting strategies. At its core, this is a zero-shot instruction, but we also include a brief chain of thought to nudge the model into a more coherent answer structure. Finally, we instruct the model to produce a structured response so we can parse it more easily.

From this prompt, any reasonably tuned model will produce something like the following, for the query “what is gpt4-o and how does it compare to gpt4”:

Notice that, from the interpretation, we can see the model “knows” what GPT-4 is but not GPT-4o. Yet, it produced pretty sensible search queries, including one individual query for GPT4-o, which is very close to what we would do if we were researching this subject on our own.

Implementing this structured response prompt in our system takes a bit more effort than just passing the prompt. The reason is we want to force the model as much as possible to produce a well-formed JSON object and, while careful prompting can get us pretty far, it is still possible for the model to deviate from just producing a JSON object, and adding some chatty messsages like “Of course, here is your JSON object:” which would make it harder to parse the response.

For this purpose, most OpenAI-compatible APIs implement something called JSON mode which forces the response to be a parseable JSON object. It won’t guarantee that you get the JSON structure you asked for (this is, in general, not solvable without modifying the sampling method) but it will guarantee that, if the model responds at all, it will be a well-formed JSON.

To take advantage of this API feature in our implementation, we will add a JSON mode method to our Chatbot class, which also skips the conversation history workflow, because we usually don’t want these messages to be part of a traditional conversation, but rather use them for one-off instructions.

In the main application loop, we simply call this method and access the queries key to find the queries.

Searching online

This is probably the easiest part of the demo. We can use any number of Google Search wrappers to obtain a list of web pages given a set of queries. In this case, I’m using googlesearch-python, which is one of the libraries with most Github stars, but feel free to experiment.

The first step is to combine all the results from the different queries into a single set so that we don’t crawl a web page twice.

This isn’t the exact code in our application, because we’d have some streamlit-specific instructions in there to print some status messages while crawling.

The next step is to crawl each of these results, skipping the ones that fail (because they are either not HTML content, or take too long to load, etc.). We use BeautifulSoup to parse the HTML and obtain a blob of continuous text extracted from every HTML node that has any text at all. This isn’t the prettiest or most robust way to parse an HTML file, especially if you want to show it to a human, but for our purposes, it works pretty well because the LLM will be able to sift through it and extract the relevant parts.

Notice that, at the same time we’re parsing the HTML, we split the resulting text in chunks of, say, 256 words, and store each chunk separately. You can probably already guess why, right?

Finding the right context

The next problem we have to solve is giving the LLM a reasonably short fragment of the relevant web pages where the answer to the user query might be. Depending on how you configure this demo, the search step might have extracted hundreds of chunks with thousands of words in total, many of which might be irrelevant. For example, you may ask for a specific event date and get a whole Wikipedia page where that event is passingly mentioned in one of the paragraphs.

To solve this problem we will resort again to the most effective augmentation strategy, retrieval augmented generation, and our old friend the VectorStore class. We will index the extracted chunks on the fly and immediately extract the most relevant ones.

The result of this snippet is a chunks list containing a small number of relevant chunks, formatted as Python dictionaries with the following structure:

The id field, computed as an incremental index, will be useful later on for printing explicit references.

Building the final response

All that’s left is formatting a proper RAG-enabled prompt and injecting the relevant content extracted from the previous step. Here is the prompt we’re using:

Notice how we explicitly instruct the model to produce square-bracketed references whenever possible. The chunks we inject in the context are pretty printed JSON objects from the previous section that contain a convenient id field.

And that’s it, all that remains is a few streamlit-specific bells and whistles here and there to get this demo up and running. For example, we add a few numeric inputs to let the user play around with the parameters of the search (how many queries to perform, how many chunks to extract, etc.)

And, if you check the full source code, we also have a few instructions here and there to make this semi-fancy layout in two columns with a search bar at the top.

Conclusions

Retrieval augmented generation is an extremely versatile paradigm. In the previous article in this series, we saw the standard approach, using just vector-based retrieval from a static text source. In this chapter, we use a generic search engine as the data source, taking advantage of the massive complexities hidden behind what looks like a simple Google search. We’re leveraging years and years of innovation in collection, storage, and lightning-fast retrieval from billions of web pages. We’re standing on the shoulders of giants, quite literally.

You can adapt this demo to any search engine that provides a text search interface. A common use case is getting all your institutional knowledge in a self-hosted search-enabled database like ElasticSearch, and using it to build an in-house Q&A bot.

By now, you should be starting to see a pattern in this integration of large language models with other traditional technologies. We will use the model to transform the user input into something structured to feed our underlying tool, compute some results, and then use the model again to produce a natural language response. This is why we talk about LLMs as a natural language user interface. It’s basically a wrapper for any traditional computational tool that adds a very convenient conversational input and output, but the tool still performs the basic computation.

This combination of LLMs with other tools helps bridge the gap between the incredible versatility of language models and the efficiency and robustness of more traditional tools, while minimizing (although not entirely eliminating) many of the inherent limitations of LLMs like hallucinations and biases.

In future articles, we will stretch this paradigm to its limits, making our LLM interact with all sorts of APIs and even produce its own executable code.

This article is part of my upcoming ebook, How to Train your Chatbot, jam-packed with practical advice and working code examples to build all sorts of coll stuff using state-of-the-art LLMS.

You can get the latest draft, plus access to all future updates, and all the Python source code in the following link, with a very special offer:

How to Train your Chatbot - 50% off

The PDF Bot

In this article, we will build our first augmented chatbot. We will use the simplest form of augmentation, called RAG (retrieval augmented generation). In this case, we will inject extra information in the prompt that might be relevant to the user query. That information will be extracted from a user-provided PDF file. This will allow the chatbot to provide concrete answers to questions about the content of the PDF file, presumably questions that couldn’t be answered otherwise.

Since we will be building on top of the basic chatbot architecture presented in a previous article, we won’t go into much detail regarding the conversation workflow, storing the history, maintaining a conversation context, etc. I’m assuming you already know how to do all of that, so we will focus on the new stuff. Likewise, instead of slowly building and refactoring functionalities, we’ll be more direct and go straight to the final implementation.

NOTE: This post may be cut in your email. I know, I know, what is this, the 80s!? Read it online here.

General workflow

The workflow of this demo will be the following:

1. The user uploads a PDF file.

2. The file gets indexed in a suitable format.

3. The user makes a query.

4. Given that query, an appropriate subset of the PDF document is extracted.

5. The model is given the context and the user query to respond.

Here is a screenshot of the final result we want to achieve.

For this workflow to work, we need to solve two basic problems:

1. how to index the PDF document such that we can retrieve the relevant fragments for a given query, and

2. how to provide that context in an appropriate prompt for the model.

First, let’s tackle those problems at a conceptual level, and then we’ll see how to code the whole thing.

Indexing and retrieval

There are many ways to store a large document for retrieval. One of the most common is called a vector store (or vector database). In this approach, we split the document into meaningful chunks of text (say, each paragraph) and compute an embedding for each chunk. Then, the user query is also embedded and compared with each of the chunks’ vectors. The closest ones represent those fragments of text that are more closely related (semantically) with the query–if your embeddings are good, that is.

This is the simplest approach, and it works good enough for most applications. However, the embedding of the query might not be the most useful key to that index. This happens, for example, if the query is something very abstract like “What are the main topics in this document”. If the document happens to contain a chunk that mentions “the main topics”, that will work. But oftentimes, these types of general questions need further processing to summarize and abstract the content of the document.

Although we will not deal with these issues in this chapter, the general workflow we will develop here can be easily adapted to more complex scenarios. We’ll say more about these extensions at the end of the chapter.

Giving context to the chatbot

Once we have identified the, say, top 3 chunks of text that most likely contain relevant information to answer a user query, we need to give that context to the chatbot.

Enter prompt engineering.

So far, we have provided the chatbot with the exact user query, but there is nothing that forces us to do that. On the contrary, we can augment that query with any additional information we have that might enhance the conversation. The simplest way to do so is to construct a prompt that contains the user query and extra instructions or information.

Here is one example:

The following is a relevant extract of a PDF document
from which I will ask you a question.

## Extract

... [chunks extracted from the vector store] ...

## Query

Given the previous extract, answer the following query:
... [user query] ...

We then feed this message as the user prompt, interpolating the right context. From the user perspective, they only asked one simple question, but from the chatbot perspective, the question comes with a big chunk of text from where to extract the answer. That is the magic of prompt engineering.

Implementing the vector store

An efficient and scalable implementation of a vector store must be able to retrieve the closest k vectors to a given input as fast as possible. Professional implementations of this functionality require advanced data structures like kd-trees and many low-level tricks to scale to search in millions of vectors.

However, for illustrative purposes, we’ll build our own, extremely inefficient and vector store. This has the advantage you won’t need to install anything to try this code, and you can always later switch it for a production-ready solution.

Our vector store is a simple collection of paired texts and numpy vectors.

To compute embeddings, we will resort to another method in the Mistral.ai API, but basically, any embedding model will do.

Note: If you notice something weird in the embed method, is because we are being a bit lazy. The embedding endpoint has a maximum token lenght, and will return an error if we send too large a batch.

We want to send the largest batch possible to save time, so to avoid computing precise token counts in the client, we simply try with all the texts, and if that fails, we split them into two subsets and try again, until we find the sweet spot.

This has the obvious downside of being slower, but for illustrative purposes, it is good enough.

For searching, we will simply compute the Euclidean distance between an input vector and the whole index.

And that’s it. In little more than 30 lines of code, we have a barebones vector store that we can use for small documents. Again, it doesn’t make sense to optimize this any further, as there are plenty of production-ready vector databases out there. Just google it.

Managing the prompt

The final piece of the puzzle is to build that augmented prompt which includes the relevant chunks for our indexed document. The actual prompt template I’m using in this demo is the following:

The extract argument will interpolate the extracted chunks, and the input argument will contain the actual user query.

Now, the first idea that might come to your mind is simply submitting the prompt (with the interpolated content) to our chatbot. And while this works, it has an ugly downside.

You see, our Chatbot implementation (taken directly from Chapter 7) does all the conversation management automatically. This means that whatever we give it as a user message, will come back to us in the conversation history. Thus, if we send the full augmented prompt as the user message, our chat history will become polluted with all those injected chunks of text. Ugly!

To solve this, we’ll make the Chatbot instance aware of our prompt engineering. We’ll supply a user template at initialization.

In the submit method, we’ll include any optional arguments, which presumably contain whatever content the template expects.

This way, the Chatbot class itself will perform the interpolation, but store in the conversation history only the clean user input.

The final application

With all the pieces in place, it’s time to build the final puzzle. Since we have dutifully encapsulated all the important functionality, our application will be a very straightforward Streamlit app.

We begin, as usual, with the preamble.

Next, we set up the file upload widget. The app will stop at this point until the user submits a file.

With a proper PDF file at hand, we can initialize our vector store. We are using st.cache_data to ensure that as long as the file doesn’t change, we won’t recompute embeddings every single time.

Next, we initialize our bot and set up the restart conversation functionality, just like in the previous chapter. Notice that this is the moment we configure the user prompt template.

The usual drill comes next: recreate the conversation history, and get the user input.

And finally, we extract the most relevant chunks and submit our augmented prompt.

And… we’re done! Our first retrieval-augmented application, in less than 100 lines of code total. Who said chatbots were hard?

Conclusions

In this post, we built our first non-vanilla chatbot application. It may seem like a small step for you, but it is a giant leap for… well, you get the point.

The crucial lesson in this chapter is how, with proper context, we can turn a standard chatbot into a powerful question-answering machine with deep expertise in any specific domain. You can use this formula to build smart assistants for your company or organization. You turn this into a research helper by feeding it papers or books (although we will build a proper research assistant). Or you can feed it with any book you’re currently reading and use it to extract insights and summaries.

One key limitation of this simplistic approach to RAG is that, since you’re embedding the user query to find relevant chunks, you might miss the answer to some question altogether if the relevant chunk doesn’t happen to have a similar enough embedding with the input. Many strategies can be used to mitigate this, and you’ll find a few suggestions summarized in the book.

The basic architecture we designed in this chapter will be the basis of much of what’s to come. The complexity will be in finding the right context and reformulating the user query in a convenient way.

If you have enjoyed this tutorial, please consider getting an early access copy of How to Train your Chatbot. It helps keep all this content free for everyone.

How to Train your Chatbot - 50% off

This article is part of my upcoming book How to Train your Chatbot. You can get early access to the book and unlock all these perks:

• Lifetime access to all future updates of the book.

• Full access to the source code with a commercially usable license.

• Online demos of all applications (soon).

• A private Discord space where you can ask anything (soon).

The early access pass is 50% off for the rest of April.

Get early access to the book

In this article, we will build a simple ChatGPT clone. We will emulate the basic chat workflow, maintaining the conversation context over several interactions.

There are a ton of ChatGPT clones out there, and ours won't be anything extraordinary, but it will lay the groundwork for future LLM applications we will build in this series.

In this article, you will learn:

• What is the difference between system, user, and assistant roles.

• How to set up a basic conversation loop with an LLM model.

• How to stream the chatbot response to simulate a typing animation.

• How to simulate the bot memory by storing the conversation history.

Furthermore, in this article, we will build critical functionalities that we'll use in future applications, such as conversation management with automatic history tracking, so we never need to write that functionality again.

NOTE: This post will be cut in your email browser. Read it online instead.

Initial setup

We will use the Python library streamlit to build the front end of our application. Streamlit is a straightforward framework for creating data science apps focusing on productivity over fanciness. This means our artistic freedom is greatly restricted, and all our apps will look the same boring way, but we will need zero HTML or CSS, and development will be a breeze.

As for the LLM, I will use Mistral.ai as the provider. They’re one of the mainstream LLM makers, just behind OpenAI, Google, and Meta in scale. They are very reliable and distribute several of their models as open source, so even though we’ll be using their cloud-hosted models in this article, you won’t be tied to their platform. You can take the same models and host it on your own infrastructure whenever you choose to.

The setup of this project will thus be simple. Our only major dependencies are streamlit and mistralai. We’ll start by creating a virtual environment and installing these dependencies.

After the initial creation, it will be convenient to dump our environment setup to reconstruct the same environment later.

Basic chat app

A streamlit application can be as simple as a Python file. Here’s the basic layout of a chat app that just echoes what you type.

Assuming this code is located at code/chatbot/app.py, we can run it with:

A browser will automatically open and show something like the following:

This is just a barebones app with no chat functionality at all. It will simply write anything you type in the input box back into the chat screen. But it will help us understand the basic application lifecycle in Streamlit.

The code above is the blueprint for a typical Streamlit app. You start by importing streamlit and setting up the page title and icon and then proceed to include any number of Streamlit statements. The code is executed top-to-bottom synchronously by the Streamlit server, and all commands are sent via WebSockets to a web app. There are no callbacks or hidden states.

If this sounds alien, don’t worry. It just means a Streamlit app works like a regular Python script: you can have global variables, methods, and classes and use them as you would in a terminal script, and it will (mostly) magically work.

The first interaction with an LLM

Ok, it’s time for the real thing. Let’s send our first message to a language model! This is easy using the mistralai package. We just need to instantiate a MistralClient and call its chat method.

First, we will need an API key from Mistral.ai. This token is a password that grants your code access to the model. Beware not to share it with anyone, or they can use the API on your behalf (and you’ll pay for it). Paste the API token into .streamlit/secrets.toml like this:

Now, we can use Streamlit’s native secret management to inject this token into our application without having to copy/paste it every single time.

Once our client is set up, we must change our chat workflow to call the LLM API and output that response.

Hit R on Streamlit to hot-reload the code and type something fancy in the input box. Maybe something like “What is the meaning of life?”. In the blink of an eye (or maybe a couple of lazy eyes), your message will be sent to the LLM API, and the response will be streamed back to your application. Voilá, we have a ChatGPT clone!

Persisting the chat state

Not so fast! You will quickly notice there is something really unsatisfying with our current implementation. Every time you send a new message, the whole chat history is cleared!

This is due to Streamlit’s simple lifecycle. Since the script is executed top-to-bottom for every interaction, every button or key pressed essentially works as if the app was just started. There is no magic here; we haven’t done anything to store the conversation, so there is no place Streamlit could get it from. Let’s fix that first.

Streamlit provides a built-in way to deal with the state that persists across executions in the object st.session_state. This is a dict-like structure that can hold whatever we want and survives re-runs (but does not refresh the page). Technically, it’s session-associated data, meaning every different user of our web application gets their own private storage. But enough technobabble, let’s see some code; it’s easier to just show how it works.

First, we initialize the history key with an empty list, the first time the app is loaded. This goes right after the client initialization.

Next, before getting into the user/assistant chat workflow, we must reconstruct the previous chat history and render all the messages.

Then, when we retrieve the new user input, we have to remember to store it in the history.

And do the same with the LLM response.

If you run this update, you’ll notice the chat history persists throughout the conversation. However, one thing is still missing. Even though the chat interface shows all the previous messages, the LLM only receives the last message. This is evident in the next screenshot, where the chatbot fails to remember information given just before.

Remembering the context

Most LLM APIs have a straightforward way to submit a full conversation history. In the case of mistralai, the messages parameter in the client.chat method expects a list of ChatMessage instances. Each message specifies the content and a role, which can be:

• system: Used to send behavior instructions to the LLM. We will see an example right away.

• user: Used to specify the user input.

• assistant: Used to specify the previous assistant input.

When you send a request to the LLM API, you can submit the previous messages (both the user’s and the LLM’s), and simulate as if the chatbot remembered the conversation. The LLM service has no actual memory, as it’s a stateless API. So, it’s your responsibility to reconstruct the conversation history in a meaningful way.

In our app, since we are already storing the conversation history in the session storage, we just need to construct the proper ChatMessage instances:

This update seems to magically add memory to our chatbot, although you need that, under the hood, there is no magic. It’s you who’s keeping track of the conversation.

Before closing, let’s take a look at the system prompt. By incorporating messages with role="system" you can guide the chatbot's behavior. In principle, this is no different from what you can do with a well-crafted user input, but some models are trained to pay special attention to system messages and prioritize them over user messages.

This means that, for example, you can override a previous instruction you gave as a user, but it will be much harder if the instruction was given as a system message. Thus, these messages often instruct the LLM to behave politely, avoid answering biased questions, etc. You can also use it to set the tone and style of the conversation.

To try this, let’s add an input box for the system messages to experiment with different prompts. By default, we’ll instruct the LLM to be nice.

And we can try it out with not-so-nice instructions. Delightful.

Streaming messages

Let’s turn the chatbot response into a nice typing animation as a final tweak. We need to do two things. First, we’ll call a different API method that returns the response as a generator instead of the whole output at once. Then, we’ll wrap that with our own generator to unpack the API response and extract the text chunks.

Fortunately, Streamlit has a nice functionality for rendering a text stream.

Go ahead and run this update, and you’ll see how the chatbot response is streamed almost in real-time (depending on how fast the API responds, of course). This has the upside that the user doesn’t have to wait until the generation is completed to see something, which greatly improves the experience.

Conclusions

And that’s it, our first ChatGPT clone! Nothing too fancy, but if you’ve never coded a chatbot before, this can look like magic. It certainly did for me!

The patterns we learned in this article will help us tremendously in the upcoming applications. Just to recap, this is what we’ve mastered:

• How to setup a basic chat workflow, complete with history.

• How to configure the chatbot with a system prompt.

• How to stream the LLM response to get a fancy typing animation.

Now that we have the basic setup ready, we can start building some cool stuff.

And that’s it for today! If you’re intrigued about LLMs, please check out my in-progress book How to Train your Chatbot. Inside you’ll find 60+ pages of content, plus the full source code for this application. The early access pass is 50% off for the rest of April.

Get early access - 50% off

This post is a preview of the first chapter of my WIP book on graphs, The Hitchhikers Guide to Graphs, written in collaboration with from .

The full content of the book will be published for free online, but if you want to support its development, please consider getting an early access pass —50% off for the remaining of April—, which gives you access to frequent updates, digital copies of all future versions forever, and a special link to get the print version at cost when it’s published.

Early Access Pass - 50% off

Ariadne Wanderlust has been tasked by Major von Veertex with mapping out the legendary underground labyrinth of Graphtopia, where a fearsome beast called the Lurker lives. Many brave adventurers had entered the maze, hoping to slay the monster or find the hidden treasure, but none had ever returned.

Ariadne was not afraid, for she was well-versed in the arcane knowledge of graph traversal algorithms! Before being appointed Chief Explorer, she taught graph search at Graphtopia University and was considered a world expert on depth-first exploration.

She knew what she had to do: walk deep into the maze for as long as possible until she hit a wall, and then backtrack her steps to find an alternative route, carefully marking the walls of the already explored paths to avoid getting stuck in a loop—all adventurers travel with a bag full of chalk, you know?

She hoped to find the Lurker’s lair and then exit before her bag of chalk ran out. However, she soon realized that the labyrinth was more complex and dangerous than she had imagined. It was full of traps, dead ends, and loops. She also heard the Lurker’s roars getting closer as it followed her scent.

She wondered if she had made a mistake by choosing this exploration technique, but it was too late to switch. Now, all she could do was hope to find the exit before the Minotaur could find her.

Modelling the labyrinth

The first step in solving any problem with graphs is to turn that problem into a graph problem!

In this case, we must find a way out of the labyrinth, so we must decide how to model it as a graph such that nodes and edges map to concepts in the labyrinth that are useful for this task. The most natural mapping, and the one you’re probably thinking about, is using edges to model the corridors in the labyrinth and nodes to model the intersections.

Here is a possible depiction of a labyrinth as a graph:

In this model, the solution to our problem –finding a way out of the labyrinth– translates into finding a sequence of nodes, each adjacent to the next, that takes us from the start S to the end E.

Let’s begin by formalizing this notion of “walking” through the graph and meet the most important algorithms that will free Ariadne from the Lurker.

Walking through a graph

The most important structure in a graph is a sequence of connected vertices. In the general case, this is called a walk, where the only restriction is that between any pair of consecutive vertices there is an edge. Let’s look at an example graph to see what we're talking about.

For example, the sequence a, b, e, d, e, c is a valid walk in our example graph, one that happens to touch all vertices. (But a walk doesn’t have to touch all vertices.) Notice that we can move over the same edge back and forth as we want, so we can extend any walk infinitely.

If we never repeat an edge (via backtracking or making a loop), then we have a trail. The previous walk is not a trail because we backtrack through (d,e). But a, b, e, d, c, e is a valid trail in our example graph because although e appears twice, we get to it via different edges each time.

If we also never repeat a vertex, then we have a path. (Some literature will use path to refer to what we call a trail, and simple path to refer to a path with no repeated vertices). In our example graph, a, b, d, c, e is a path that happens to involve all vertices.

If the path loops over from the final vertex back into the first one, like a, b, d, c, e, a, then we call it a cycle.

Graph traversal

The simplest graph procedure involving some notion of “walking” is graph traversal. This task involves enumerating all nodes in a predefined order by moving through the edges. That is, we don’t want to simply list all nodes; we want to order them in a way that uses the graph structure so that subsequent nodes are connected.

There are two basic graph traversal algorithms: depth−first search (DFS) and breadth−first search (BFS). Both algorithms are very similar and will produce a full enumeration of a graph. DFS and BFS differ in how they prioritize being eager versus being comprehensive.

Abstract traversal

We will begin by defining what our abstract notion of “search” looks like. To keep things simple, we assume that a search algorithm provides a single method traverse that simply enumerates all edges in the order in which they are discovered.

The nodes method is just a thing wrapper around traverse that yields the nodes instead of the full edges.

Why make this a class? Isn’t this just a method? Well, it’s a bit of a mouthful at the moment, for sure. But later, as we explore many search algorithms, we’ll want to compare different strategies. That’s when the search interface will shine.

Plus, this abstract method allows us to implement a very common search pattern that we’ll’ reuse over and over: searching for a specific set of nodes. (Including a single node.) We can have the general−purpose case that matches any node with a given property and the special case when we need to find one particular node −− e.g., like the exit of the labyrinth.

With this code in place, we’re ready for some actual search algorithms.

Depth−first search

As the name implies, depth−first search is a graph traversal algorithm that prioritizes going as deep as possible as fast as possible. In our labyrinth analogy, this means turning right until a dead end and backtracking to the last unexplored intersection.

More precisely, DFS starts at an arbitrary root node in the graph and then jumps to one of its neighbors, continuing the traversal from there. You can select which neighbor to visit by a random choice, but most commonly, one simply defines an order—e.g., the order in which neighbors are listed in the adjacency list.

In a practical scenario, this could mean always trying to move south, east, north, and west. Of course, you must track which nodes (or intersections, in this case) you have already visited. Otherwise, you can easily get stuck in a loop.

This is what DFS looks like in our sample graph that models the labyrinth problem.

Programming DFS

Let’s implement DFS! The easiest way is via recursion1: we start at the root node and recursively visit all non−visited neighbors. To make sure we don’t get stuck in a loop, we keep track of the explored nodes throughout all the recursive calls.

Each iteration returns the edge (x, y), where y is the current node under consideration, and x is the “parent” node −− that we must also keep track of during recursion.

Here is the full implementation:

As it’s common in recursive methods, we have a public “portal” method traverse that exposes the public arguments, which in turn defers to a private implementation _dfs that takes any additional arguments we need for bookkeeping.

Finding your way out

Armed with the arcane knowledge of depth−first search, it is now trivial to exit the labyrinth (provided there is indeed a way out). We just need to run DFS in the start node and, eventually, we will reach the end node. We might have to backtrack once or twice if we are unlucky to pick to wrong turn, but as long as we keep track of every node we visit, and make sure not repeat any of them, the way out is guaranteed to be found.

Computing paths

While knowing that a goal node exists is useful, we often want to find the actual path that takes us there. Fortunately, our abstract Search strategy can implement this operation easily using the parent from the tuple (parent, node) available in each iteration in the traverse method.

To quickly compute paths, we can define a simple structure (Paths) that stores a reference to the parent of each node found during search. With this information, the path between our origin vertex and an arbitrary destination is easy to compute by following the links backwards. That is, we start at the destination node, and iteratively add the parent node to a list, until we find the origin node. Then, we simply reverse the list.

Paths are origin−dependent (!)

You’ll notice we don’t require an origin parameter in the Paths.path method. That is because this structure holds paths precomputed from a fixed origin node, that is, the node from which the search algorithm started.

If you need to precompute paths for arbitrary pairs of nodes, there is little you can do other than using a Path instance for each origin node.

With this structure in place, we can add a method to the Search class to compute all paths for a given graph and origin.

Analyzing DFS

Before finishing, let’s turn our attention now to the analysis of this algorithm. In computer science, we are often interested in answering a few critical questions for every algorithm:

1. Does the algorithm always work?

2. How fast does it work?

3. How much memory does it need?

4. Is there any better algorithm?

The first question asks about the correctness of the algorithm. An algorithm is only useful if it always works −− or if, at least, we can determine beforehand whether it will work or not. In the case of DFS, we can claim with absolute certainty that, if there is a path from origin to destination, the algorithm will eventually find it.

¿Why? We can give an intuitive justification of correctness for DFS as follows. Since we never repeat any vertex, and we explore all neighbors of the vertices we visit, we must visit every vertex, that is reachable from the origin, eventually.

Now, there is one massive caveat with DFS. You have no guarantee the path you find is ₜhe shortest path_ from the origin to the destination. Because DFS walks down a path as long as possible, and then never revisits those nodes, you can actually discover the destination first by going throught the long way, instead of taking a shortcut.

The second and third questions ask about the performance of the algorithm. If we want to compare two algorithms for the same problem, there are almost always at least two obvious dimensions: running time and memory requirement. We usually want the fastest algorithm, provided we have enough memory to run it, right?

Now, for reasons beyond the scope of this book, it is almost always useless to think in terms of actual time (like seconds) and memory (like megabytes), since the exact values of those properties for any given algorithm will depend on many circumstancial details like the speed of your microprocessor or the programming language you used.

Instead, to compare two algorithms in the most abstract and fair possible way, we use a relative run−time and memory, that just considers the “size” of the problem. In most cases in this book, the “size” of our problem is something we can approximate by the total number of edges and vertices in our graph.

Thus, we can formulate questions 2 and 3 in terms of, given a graph of n vertices and m edges, how many steps of DFS do we need? You can probably suspect the answer by looking at [@fig−labyrinth−solve] and counting the numbers. You will notice the answer is just the number of edges, since DFS explores the whole graph, and never repeats any path. In general, most search algorithms in this book will take a time that is proportional to the total number of vertices and edges in the graph.

Regarding memory, we could assume we also need something proportional to the total size of the graph, since we need to mark every visited node, right? Well, that is true only if there are loops in the graph. But if you know beforehand there are no loops, then once you hit a dead end and backtrack, you can forget that whole branch up until the bifurcation. This means that, on average, you will only need enough memory to remember as many nodes as there are in the longest possible path in the graph.

Finally, we can ask if there is any way to discover the exit faster. The general answer is “no”, unless we know something special about the graph. That is, in a general graph, without any extra knowledge, you have absolutely no idea where the exit (or the destination nodes in general) will be, so at worst you’ll have to explore the whole graph.

However, as we will see a few chapters down the road, in many ocassions you do know something extra about the graph—e.g., when driving around a city, you have some sense of which direction your destination is. In these cases, we can often speed−up the search massively with some clever strategies.

Final remarks

Depth−First Search is a cornerstone of graph search. Almost all algorithms in this book will either include it as an explicit step, or use it as a building block.

Besides being extremely simple and elegant to implement, DFS is also the easiest to adapt to a real−life situation. Unlike most other graph search algorithms, DFS only moves to adjacent nodes. So, if you are an agent exploring a graph−like structure in real−life—just like Ariadne—DFS is what you would most likely use.

The main caveat of DFS, as we already saw, is that you cannot guarantee that the first time you discover a node, you did so via the shortest path to the origin. In fact, it is easy to come up with examples where you actually reach your destination through the longest possible path; it all depends on how lucky you are selecting which neighboor to visit next.

Next time, we will look at an alternative way to explore a graph that guarantees shortests paths, but it requires that you can teleport to any given node. If you’re want to read the chapters as soon as possible, consider getting an early access pass.

Early Access Pass - 50% off

Building on the success of the last lesson on making a bot to chat with a PDF file, today we’re reusing some of those ideas to build a customer service bot that can reply to user queries using a combination of profile information and a general knowledge base.

This is a toy example to illustrate a widespread use case. Imagine you have a SaaS product with a FAQ guide containing information such as pricing and features per plan, discount offers, a general description of your service, regional availability, etc. Usually, your service users need to read the guide and find answers about their specific situation: which plan they’re using, what country they’re based on, etc.

Instead of making your users parse the entire FAQ, ignore everything irrelevant to them —such as enterprise offers— and find the relevant information that answers their specific question, you can use an AI-powered chatbot.

This chatbot will answer a user query with information from the FAQ but —crucially— inject into its context details from the current user’s profile, so the answers will be tailored to this user.

If you want to check what that looks like in a toy situation, here’s an online demo you can play with, and here’s a short video of a possible interaction.

The whole source code for this lesson is available on Github.

Prerequisites

This lesson is somewhat advanced, so I expect you to be comfortable with basic Python and building simple apps in streamlit, although most of the streamlit code will be self-explanatory.

You must also know how to set up a basic Python project using a virtual environment and install dependencies from requirement files. If you need help with any of these topics, check out The Python Coding Stack • by Stephen Gruppetta and Mostly Python for plenty of beginner-friendly articles.

You don’t need prior knowledge of using LLMs or vector databases or building chat-based apps.

This post will be cut on your email, so read it here online.

Making a Customer Service Bot

This lesson is somewhat advanced, so I won’t explain the code step by step. Instead, I will give you a high-level description of the app's architecture and then zoom into those crucial implementation details.

My purpose, as usual, is that you understand how each functionality is implemented and how they fit together, so when you read the actual code, everything clicks into place. Thus, I won’t go line by line but rather look at the main blocks in the order that makes understanding the app workflow easier.

At a high level, this is a typical streamlit application with a simple layout. It has a sidebar where the user can upload a PDF file, and the main content is a container for chat UI elements.

The workflow is as follows:

1. Initially, the FAQ guide is indexed question by question and stored in a flat in-memory vector index using faiss.

2. Then, the user asks a question.

3. The user query is embedded, and the top-3 most relevant FAQ entries are extracted.

4. A custom prompt is built with the FAQ entries and the current user metadata.

5. Using mistralai, a response is obtained from an LLM.

6. Go back to step 2.

But the devil is in the details, so let’s dive in.

Indexing the FAQ guide

Let’s assume the FAQ guide is a collection of questions and answers covering most users’ needs. We want to select a subset of 3 relevant questions for a given prompt. We can use the magic of vector embeddings for that.

Our example FAQ guide is a simple markdown document where each question starts with `#`. Go ahead and read it. It’s hilarious.

First, we will split the FAQ into chunks of a manageable size —in this case, one chunk per question.

Once loaded and split, we use the faiss vector library to create a simple flat in-memory index. We can add a bit of streamlit magic to render a progress bar as this indexing takes place.

The @st.cache_resource decorator ensures that we only create one index per execution of the application.

The embeddings are computed with a handy function that we will see later.

Answering questions

Once the FAQ guide is indexed, we can take user questions. We will use Mistral AI as our supplier of LLMs because they provide really powerful open-source language models that challenge even ChatGPT.

Here is the monster function to reply to a specific user query.

Let’s break it down.

First, we embed the question and query the indexed FAQ to retrieve the top 3 most similar documents. These correspond to 3 questions (and their answers) that are linguistically close to our query, so there is a good chance one of them has the right answer. Here’s the embedding function:

Then, we verbalize the user data (e.g., username, role, credits, plan, etc.) to feed it to the LLM. In this case, we simply produce a line-by-line text in the format key: value, for example:

Username: Neo
Plan: free
Credits: 100
Profession: Engineer

Finally, we combine these two pieces of information with the actual query in a custom prompt and send it to the LLM. Here’s the prompt:

We need a Mistral client correctly configured (assuming you have your API key in the secrets file).

Chat management

The final piece of the puzzle is the actual chat workflow. We will use Streamlit’s chat widgets with some tricks to simulate a conversation.

The gist of the chat management problem is that since Streamlit is stateless, we need to rebuild the entire conversation on each execution. However, we want to show the typing animation only for the last message and simply output the previous history.

We will store all messages in a session state list to achieve this. The add_message function displays the current message (with a nice typing animation) and saves it for later.

The msg argument can be one of two things: a regular string or an actual asynchronous response for Mistral. In the first case, we wrap it in a function that will turn it into a stream of words with some delay in between to simulate it is being typed.

In the second case, we just need to unwrap the generator and yield the actual content:

Putting it all together

Now that all the pieces are in place let’s see how the application workflow works.

First, we initialize our session state and re-write the chat history:

Then, we compute (or simply restore from the cache) the FAQ data and index and get the user data and query.

If this is the start of the conversation, we want to output some information.

And since we want the first message from the LLM to be something informative, we query it with this initial request:

And finally, we take the user query and compute a reply:

And that’s it! In little less than 250 lines of code, you have a chatbot that can answer custom queries combining user data with a general knowledge base.

Moving forward

This simple example can be extended to any length for the FAQ guide (or any other content guide) and any complexity of the user data.

However, if the user data becomes too large, it won’t all fit in a single prompt. You’ll need to apply clever filtering to select the subset of data relevant to the query.

In this case, you can treat the user data as another document, split it into chunks, index it, and let the embeddings do their magic.

Other limitations that might give you ideas for improvement are:

• Only the last message is sent to the LLM, so even though the whole conversation is displayed constantly, every query is independent. That is, the chatbot has no access to the previous context. This is easy to fix by passing the last few messages on the call to client.chat_stream(...).

• There is no caching of queries, only embeddings. The same query will consume API calls every time it is used. This is relatively easy to fix by caching the query before submission, but you cannot simply use st.cache_* because the response is a stream of data objects consumed asynchronously, and the same query can have a different response per user/document.

• Only three chunks are retrieved for each query, so the model will give an incorrect answer if the question requires a longer context.

• There is no attempt to sanitize the input or output, so the model can behave erratically and exhibit biased and impolite replies if prompted with such intention.

And that’s all for today. As usual, let me know your opinions about these coding lessons. Are they too specific, too fast, too technical, too dull?

Leave a comment

And if you think this is worth it, consider leaving a like and sharing this post with a friend.

Share

Today’s lesson is about building a bare-bones app to answer arbitrary questions from a PDF file. We will use mistral.ai as the provider of language models, which has some incredibly powerful open-source LLMs1.

Here’s a quick look at the final application, answering questions on the latest draft of my upcoming book The Science of Computation.

The full source code (minus Mistral API keys) is located here.

Note: This post will be truncated in your email. View it online here.

Prerequisites

This lesson is somewhat advanced, so I expect you to be comfortable with basic Python and building simple apps in streamlit, although most of the streamlit code will be self-explanatory.

You must also know how to set up a basic Python project using a virtual environment and install dependencies from requirement files. If you need help with any of these topics, check out and for plenty of beginner-friendly articles.

You don’t need prior knowledge of using LLMs or vector databases or building chat-based apps.

Making a Chat with your PDF app

Since this lesson is for advanced readers, I won’t explain the code step by step. Instead, I will give you a very high-level description of the architecture of the app (which you will see is extremely simple), and then I will zoom into those implementation details that are crucial, leaving aside the boilerplate code.

My purpose is that you understand how each functionality is implemented and how they fit together, so when you read the actual code, everything clicks into place. For this reason, I won’t go line by line either but rather look at fundamental blocks in the order that makes understanding the app workflow easier.

At a high level, this is a typical streamlit application with a simple layout. It has a sidebar where the user can upload a PDF file, and the main content is a container for chat UI elements.

The workflow is as follows:

• The user uploads a PDF file.

• Using the pypdf module, we extract the plain text and split it into overlapping chunks.

• Each chunk is transformed into a vector, using Mistral’s embeddings via the mistralai module.

• The embeddings are stored in a flat in-memory vector index using faiss.

• Then, a query/response cycle is run, using one of Mistral’s LLMs, with a custom prompt that injects the most relevant chunk of the document for a given user query.

But the devil is in the details, so let’s dive in.

PDF processing

Let’s start by analyzing the PDF processing step. Although this is almost the last piece of code written, it is the starting point that triggers the whole workflow.

As is typical in Streamlit, we use a widget to let the user upload a PDF file and store the resulting object in a global variable.

However, since we want to do batch processing with it, and Streamlit runs this script every time something changes, we will use a callback that is only invoked when a new file is uploaded via the on_change parameter. (Notice we name the widget with a custom key, this will be important soon.)

The build_index function gets called whenever the PDF widget changes, that is, when the user selects a new file.

The first part of this function is straightforward. We take care of the case when there is no PDF uploaded (the use of session_state.messages will be clear later on), and then proceed to extract the text from the PDF.

Notice we obtain the PDF file from the session_state.pdf_file key, which is exactly the key we used when naming our PDF upload widget. This is one of the ways you can connect different widgets in Streamlit without using global variables.

The last two lines of this first part of the function store the extracted text in the session state dictionary for later use and output a helpful message indicating the size of the uploaded document. This is mostly so the user knows something is happening, as the whole indexing can be slow.

Next comes the actual indexing part.

First, we split the text into chunks of 2048 characters, with an overlap of 1024. This is the crudest way to split a text document into meaningful parts, but it works as a proof of concept. The overlap ensures that if we (almost certainly) split an important sentence in half in one chunk, we will get a second chance to store it in full in the next chunk.

After the chunks are created, we will compute an embedding for each. The embed function (which we will see in a minute) calls the Mistral API and returns a vector (in the form of a list of float numbers). We stack them all together using numpy and create a huge matrix that has one row per chunk.

Finally, we use faiss’ IndexFlatL2 class to store all the embeddings in an efficient data structure we can query later. We then store both the index object and the chunk list in the session state because we will need to recover the actual chunks using this list.

Answering questions

The question-answering part is the core of the application. We implement this functionality in a function reply that answers a single query. Then, we’ll see how this all fits together with the app workflow.

Here it is:

The first two lines compute the query embedding, which we use to find the closest chunk in the index in the next two lines.

Then, we use the Mistral API to query the LLM with a single message (the user query), but —here is where the magic happens— that query is wrapped in a custom prompt that injects the most relevant chunk of text (according to the embeddings at least) along with some instructions for the LLM.

Here is that prompt:

Because LLMs are pretty insane, this actually works most of the time. The LLM will see a block of 2048 characters closely related to the query and will almost certainly manage to extract the relevant part from that blob of text to answer the query.

Two details are still missing from this part. One is the instantiation of the Mistral API client, which is pretty straightforward, provided you have the right API key in the Streamlit secrets file:

The @st.cache_resource decorator avoids creating a new API client on every run.

The second implementation detail missing is the embed method, which basically just calls the Mistral API. We add the @st.cache_data decorator to save us from recomputing embeddings for the same blocks of text over and over.

Chat management

Now that we know how to reply to one user query, let’s see how the actual conversation workflow works. We need to solve two things: persist the conversation history despite Streamlit’s stateless execution model and simulate the typing animation. The following function takes care of that:

The key pieces of this functionality are the following. We use st.write_stream instead of the typical st.write to simulate the typing animation. This method receives an iterator of printable stuff (e.g., characters). In the case of the actual LLM response, we will receive an iterable of responses. The following function (used in reply) unwraps the response content, turning the LLM stream into a simple string iterable.

For our system messages (like the welcome screen), we must wrap that string in a function that yields each character at a time.

The last part of the add_message function simply stores the message just streamed in a session state list so that, in the next execution, we can rewrite all previous messages and simulate a persistent conversation.

Putting it all together

Now that we understand all the puzzle pieces let’s see how they fit into the main application workflow. The first step is to initialize the session state variable to hold our chat history (if this is the first run) and then write that history back into the chat container every other run.

Next, we show an introductory message only if we haven’t done the PDF processing yet (remember we store the PDF text in the session state when we do). These two messages will not be stored in the chat history so they will be replaced once we upload the PDF.

Nex comes the PDF upload widget we’ve already seen. Once we have a PDF uploaded, we take the user query.

We also retrieve the IndexFlatL2 from the session state for use in the next fragment.

And finally, we send the query to the LLM to compute the reply. However, the first time, we will add a custom query that simply asks for a document summary.

Final remarks

And that’s it! In a little more than 200 lines of code, you got yourself a fully functional, although simple, app to chat with an arbitrary PDF file. All thanks to the magic of LLMs and the simplicity of the streamlit framework,

There are some obvious limitations in this implementation, though. Here are some of the most relevant, which can give you ideas to improve it.

• Only the last message is sent to the LLM, so even though the whole conversation is displayed constantly, every query is independent. That is, the chatbot has no access to the previous context. This is easy to fix by passing the last few messages on the call to client.chat_stream(...).

• To save resources, documents with more than 100 chunks will error. This number can be changed in the source code.

• There is no caching of queries, only embeddings. The same query will consume API calls every time it’s used. This is relatively easy to fix by caching the query before submission, but you cannot simply use st.cache_* because the response is a stream of data objects consumed asynchronously, and the same query can have a different response per user/document.

• Only one chunk is retrieved for each query, so the model will answer incorrectly if the question requires a longer context.

• There is no attempt to sanitize the input or output, so the model can behave erratically and exhibit biased and impolite replies if prompted with such intention.

• All LLMs are subject to hallucinations, so always double-check your responses.

And now we’re done! Feel free to clone and modify this project to your needs, and please let me know if you enjoyed this lesson, especially if this format is engaging and informative for you. All feedback is appreciated!

Leave a comment

The lessons are 100% free but really hard to make. I would appreciate a like and a share if you think they're worth it :)

Share