Agents as the Final Frontier in Large Language Modeling

Bringing LLMs to life with beliefs, goals, and intentions.

This article is part of my in-progress ebook How to Train your Chatbot, jam-packed with practical advice and fully working applications on how to use LLMs for building all sorts of cool stuff. You can get early access for half the usual price.

So far, we’ve connected LLMs with other systems, extending their knowledge and enhancing their capacity to solve complicated problems by leveraging different tools. But all the tasks we can solve so far are essentially single-shot, requiring little to no planning once the task is well understood and the right tools are selected; regardless of how complex the tasks we’ve seen, the number of steps they require is fixed from the beginning.

To handle the most complex tasks, we must give LLMs the ability to pick and use arbitrary tools and actively reason about the task while performing it, incorporate new insights, and change their minds during the solution process. LLMs must become active agents rather than passive tools. This is what the agent paradigm is supposed to unlock. Agents are not a new thing in AI or computer science in general, though. In fact, it is one of the most important software design paradigms since the primordial ages—back when Turing was thinking about German codes.

In computational terms, an agent is just a software component that acts autonomously, interacting with a (computational) environment and constantly making choices to further some goals. The difference between traditional software and agent-based software is that, in traditional software, all changes to a system are in response to some user input: that is, the user is the active element. In agent-based software, the software itself is active in the sense that it is always making adjustments and producing effects on the computational system, regardless of whether there is a user.

For example, consider the difference between a passive AI assistant, like Alexa or Siri, and an active one. In the former case, the assistant is essentially dormant until you make a request. Then, it turns on, parses your query, and takes any actions needed to fulfill your request before going back to sleep. All interactions are started by you, the user.

In contrast, an active AI assistant would always be processing stuff, reading the news, scanning your emails, searching in Google, etc. It could forward you any interesting links it finds online and update your calendar as new tasks arrive in your inbox. But, more importantly, it could be working on a specific task for as long as necessary without your continued input.

Now, you can stretch the definition so that any piece of software looks agent-based. Your operating system is always on, doing stuff, ready to ping you when there’s a problem. It has a goal—keeping the system running—and it can interact with an environment—hardware drivers, peripherals, hard drives, etc.—and exert actions to make changes to the system. But this is seldom a helpful way to think about your operating system, mainly because its goals are pretty straightforward, and the actions it can perform are very restricted.

When you give a software agent long-term goals without immediately apparent steps, things turn interesting. The agent must then observe the environment, gather information, maybe ask you for clarifications, and make a plan to further those goals. Moreover, as it discovers new information, it may refine those goals, split them into lower-level subgoals, and generally navigate around a complex decision-making process that wasn’t explicitly programmed or even obvious to an external observer from the beginning.

Adding language models to the mix unlocks exciting new possibilities. Now, your agent can not only communicate with you in natural language but also talk to other agents easily—there is no need to define formal communication protocols or APIs—and even reason and argue with itself in natural language.

---

---

Anatomy of an LLM-powered agent

An LLM agent is, ultimately, just a plain old AI agent with some linguistic capabilities. So, let’s review the general definition of an agent in the AI field and explore some of the basic agent architectures invented.

In its most straightforward conception, an agent is a software component embedded within a larger computational system we call the environment. This environment can be anything from a computer simulation to the whole Internet, including other agents (computational or otherwise). The agent always has some goals—which can be as abstract and ambiguous as you want—that it must further by performing some actions. For this purpose, the agent has some sensors by which it can obtain information from the environment and some actuators that allow it to exert changes in the environment. Finally, the agent has some internal reasoning process to decide which actions to perform given the current environment state.

This picture is complete from the outside, but we can further refine it if we look inside the agent. Here is where literature is—quite literally—overflown with variants of “standard” agent architectures. So, I won’t attempt to cover the majority of variants or even be too formal. Instead, I’ll explain the main ingredients in most used agent architectures, using a casual nomenclature.

The most basic agent that does some form of reasoning needs to do two separate things: keep track of the state of the world (which may include its own state) and then reason about that world to decide which actions to take. This reasoning can be cyclical, updating its own state with new information inferred from the previous state. We can thus come up with a sensible basic agent architecture.

This architecture is sufficiently detailed for us to discuss different variants while being sufficiently abstract to allow many different use cases and implementations. For example, we haven’t explicitly defined what goes on in the internal representation of the world: it could be anything from a structured set of values to a list of natural language facts to a full-blown database. We also haven’t specified what sensors or actuators actually are, which depend heavily on the concrete use case.

Adding natural language

So far, we haven’t used the fact our agent is LLM-powered, and for good reason. The basic agent architecture we just defined is agnostic to implementation details. Adding an LLM to the agent architecture just means some or all of the modules are implemented using language models: that is, with prompts, RAG, function calling, and everything else we’ve seen so far.

First, perception may be LLM-based: your agent can receive information from the environment in the form of natural language documents. Likewise, the actuators can be LLM-based, e.g., via function calling. More importantly, the reasoning process and internal representation may also involve language models. The internal representation can be a straightforward collection of natural language claims about the environment, which the agent updates and refines via reasoning. The reasoning can be guided by some clever prompting and thus occurs entirely within a linguistic framework. Finally, the environment can contain other language models or be implemented as a language-based simulation with clever prompts.

Use cases for LLM agents

The most obvious use case for an LLM-based agent is some sort of always-on assistant, your personal Jarvis if you will. Unfortunately, most AI assistants today are more tools than agents. They turn on after a user command, do their thing, and return to idle. You start all interactions between you and Siri, Cortana, or Google Assistant.

Moving towards more agentic behavior, we should expect an AI assistant who has long-term goals—e.g., keeping you informed or keeping your calendar well organized—and is always consuming information and taking actions to further those goals. So here are a few possible, very broad use cases for such an agent.

The planning agent

This is one of the most useful and yet unexplored use cases. An agent connected to all your input sources—email, work and family chat groups, etc.—collecting all information about upcoming events and tasks. Its goal would be to help you maintain a healthy work-life balance by keeping your calendar current. The agent could reply to a meeting email and ask for a reschedule if it knows you’ll be busy that day. Now imagine everyone has such an agent, and they all negotiate with themselves and find the optimal schedule for the group meeting.

The research agent

This is an agent whose goal is to answer a complicated, nuanced research question. It can be related to your work, like mapping a business opportunity, or more akin to a typical scientific research question. The agent would search online for relevant sources, extract insights, update its internal representation, and generate new follow-up questions, building a web of interconnected information for a given domain.

The coding agent

This is the software engineering holy grail: an agent that continuously scans and improves an existing codebase, perhaps adding unit tests, comments in code, or refactoring existing functions. Additionally, this agent could pick low-complexity issues (like obvious bug fixes) and attempt to solve them automatically, even crafting a pull request when all test cases pass. Furthermore, it could revise existing pull requests and suggest improvements or general comments before a human revisor takes over the final merge.

The gossip agent

This is more tongue-in-cheek, but imagine an agent monitoring all your social accounts: Facebook (who has that?), Twitter/X, LinkedIn, your friends’ WhatsApp group chats, etc. Every time some event happens—e.g., someone announces they got a job or are traveling somewhere, etc.—the agent updates its internal representation of your social network. And periodically, or whenever something significant changes, it will send you updates on who’s doing what or going where.

Beyond the single-agent paradigm

The above are reasonably straightforward applications of the agent paradigm using LLMs. While the devil is in the details, and implementation is often much harder than first thought, I can see all of the above examples working in the near future, at least within reasonable constraints. However, single-agent architectures are just scratching the surface of what this paradigm can bring. The real deal is in multi-agent environments.

In a multi-agent setup, different agents coexist, collaborating and/or competing to achieve common or distinct goals. The power of multi-agent architectures is that a complex problem may be very hard to model with a single agent but feasible when subdivided into distinct goals and skills. Take, for example, the coding agent scenario mentioned before. Instead of a single agent taking all these tasks one at a time, imagine multiple agents working in parallel, some optimized to fix bugs, others to comment code, and others yet to revise, add tests, make suggestions, etc.

And this is more than just N different agents working in parallel on a codebase. These agents may have to coordinate to solve a big task. For example, maybe one “software architect” agent will take a complex issue and make a plan, dividing it into subtasks for different features. Then other “implementation” agents will work out the simpler tasks and do the actual coding.

Furthermore, you can have gatekeeper agents that revise code and emit some sort of qualitative evaluation. They can even ask questions, and the implementors must convince them of their solution. A lot of old and recent research suggests that, when combined, multiple agents can solve problems that none of them could tackle on their own.

Why would this work better than just a super coder LLM? You can consider it an advanced form of ensembling mixed with self-reflection, which are, in turn, advanced forms of chain-of-thought. Combining multiple LLMs with different tasks (i.e., different prompts) in iterated interactions gives them more chances to revise and regenerate, which translates into more computational power. At the same time, we’re leveraging the ensemble effect of pooling different solutions to the same problem, which tends to produce better solutions on average as mistakes are smoothed out.

Symbolic learning in LLM agents

Putting a bunch of LLM-powered agents to work together is neat, but we can go further. The next step is to make our agents learn from their interactions with the environment.

In the classic agent architecture we’ve discussed so far, no explicit learning is happening inside the agent. To some extent, you can think of updating the internal representation as learning about the environment. Still, there is no learning about the strategies the agent can use. The reasoning module, however, is implemented, has no way (at least so far) to update itself so that, given the same environment, it could perform better in future occasions instead of repeating the same mistakes.

The standard solution to this problem is reinforcement learning. Assume some optimizable inference architecture inside the agent’s reasoning module (e.g., a neural network) and use feedback from the environment to update its parameters. But what if we could achieve something similar using a purely linguistic approach?

One way to achieve this is to give the agent the ability to reason about its own performance and generate hypotheses—natural language claims—describing what it has learned. These hypotheses can be stored in a private knowledge base (separate from the environment description) and retrieved during reasoning.

In the simplest case, this is a long-term memory of what the agent has tried in different situations and whether it has worked. In future situations, a RAG approach could provide examples of things that have worked before. However, this can be refined further by progressively asking the LLM to produce high-level hypotheses that abstract and summarize similar situations, eventually building operative knowledge more general than any specific past situation.

What’s incredible is that this approach has improved reasoning and problem-solving skills in domains as complicated as answering medical questions. In a recent paper, researchers put agents in a simulated hospital environment and generated thousands of cases of patients with different conditions. The whole simulation is run by linguistic agents, from patients to nurses to doctors, who interact with each other and decide on a treatment. External rules (from a medical knowledge base) simulate whether the treatment is effective or not. After each case is completed, the doctor agents write a detailed report, which is stored in their internal knowledge base.

After ten thousand or so iterations, the doctor agents had built a case database that contained distilled knowledge about every possible rundown of diseases, symptoms, treatments, etc., they had seen. This is equivalent to what medical residents see in two years of training. And incredibly, when provided to a GPT-4 single-shot question answering bot, this database improved its results in standard medical question benchmarks by almost 10%. In a sense, the LLMs taught themselves medicine by interacting with each other in a simulated medical scenario.

This is akin to reinforcement learning but is parameter-free, 100% transparent, and explainable. Instead of updating some opaque weights in some neural networks, your LLM-powered agent is learning like humans: by analyzing its actions, taking notes, and constructing hypotheses. While highly experimental, this is one of the most exciting use cases for LLM agents I’ve seen recently.

Caveats and limitations

As in everything related to LLMs, you must beware of hallucinations and biases. Ensembles, in general, and agents, in particular, are more robust to random errors because of their distributed nature. This means that unlikely hallucinations that could be deal-breakers in single-LLM scenarios might get smoothed away when you pool together multiple LLMs, taking turns to make choices.

However, biases are not random but systematic errors. Using the same model to power all your agents means they are all susceptible to making the same judgment mistakes over and over. There is no algorithmic solution to reducing biases that doesn’t attack the original problem: biased data.

Another major caveat of LLM agents is that they are significantly more costly compared to a single zero-shot or a simple chain-of-thought prompt. Your agent is always running, which means you’re constantly calling the LLM API, even if there is nothing new to be done. This underscores the need for in-house, open-source language models that can be cost-effectively scaled to these heavy-duty scenarios.

Finally, agents in AI have a long history of being abused as an overengineered solution to problems that could otherwise be solved with more direct methods. Do your own experiments and test whether simple prompts or more straightforward augmentation techniques are sufficient for your use case before building an entire simulated environment.

Conclusions

The agent model is one of the most exciting paradigms in artificial intelligence, with a long and rich history of architectures and approaches. It is also one of the hardest to implement, at least until now, because it always required far more intelligence than was possible with previous approaches. However, with LLMs, agents are seeing a rebirth as one of the most promising approaches to building reliable and truly intelligent AI solutions.

I am, however, fairly biased. This is one of my dearest research topics, and I may be over-excited. So take everything I said here with a grain of salt. Modern agent architectures are still brittle and hard to deploy on a large enough scale for their distributed strength to be noticeable. Still, they hold great promise and are one of the few paths forward that I believe may take us all the way to general-purpose AI.

We will build some agent-based simulations in Part 3 of the book, especially for the final chapters. Due to their complexity and cost, we won’t spend as much time building agents and simulations as I would like, but you’ll get a taste of how powerful this paradigm can be in chapters 24 and 25.

If you want to learn more, grab your early access to How to Train your Chatbot in the link below.