The Year of AI

As 2023 closes in, let's take a brief look into how we got here, and what is waiting right ahead for us.

The year 2023 is undoubtedly the year of Artificial Intelligence. The release of ChatGPT was the iPhone moment of AI. While it isn’t necessarily a fundamental breakthrough in the scientific or technical advancement of the field, it undeniably marks the inception of the widespread of AI. It has already had a significant cultural impact. It has reshaped public perception and discourse surrounding AI, bringing awareness of its potential and ability to influence our lives.

For a long time, Artificial Intelligence has been in the collective imagination, from its portrayal in science fiction to many mildly successful attempts at practical applications. However, with ChatGPT's release and the subsequent surge in large language models and chatbots, AI has finally become a fundamental technology for the general population, even if it is nascent.

Reasonably enough, many skeptics –myself included on occasions– question AI's authenticity and transformative potential in its current form beyond the hype and excitement of a shiny new toy. Debates on whether AI is merely an illusion or a revolutionary force happen daily across all traditional and digital media.

In this last issue of Mostly Harmless for 2023, we will review the evolving promises and expectations of Generative AI, highlighting the essential challenges and unresolved issues necessary for the technology to realize its intended potential.

Approaching the subject with a healthy balance of skepticism and optimism, I will critique the current state of generative AI from the perspective of a deeply optimist machine learning researcher who loves the field. I will re-examine the most impactful milestones that brought us here and assess the challenges and obstacles ahead.

By the end, I hope you share with me the excitement of being in the early days of what could potentially be the most important invention of our civilization, with the necessary suspicion and open-mindedness to understand the path forward isn’t guaranteed.

---

---

The road so far

Let’s start by looking back at the last few years in generative AI, from the inception of the transformer architecture to the release of the first widely useful AI application.

When discussing generative AI, we can address different domains: language/text, images, and audio/music. Arguably, the most impactful advancements are in the text domain due to the emergence of large language models. The other domains have also seen incredible progress with the development of, e.g., diffusion models allowing for controllable text-to-image generation and realistic text-to-speech and speech-to-text conversion.

Thus, I want to specifically focus on language, first because it’s my area of expertise, and second, because the majority of economically valuable jobs projected to be automated in the near future will be due to the existence of language models. While the other domains certainly have exciting applications in art, marketing, video games, and content creation, the potential reach of very powerful language models with real language understanding surpasses that of generative sound, music, and image combined.

The reason is simple. Language is the primary means of human communication, interaction with the social environment, collaboration, and task-solving. Natural language is frequently used to describe tasks and their solutions. Therefore, solving natural language understanding and developing a machine capable of comprehending instructions, following them, and communicating back in a natural language could potentially be the most transformative technology we could ever invent. We’re still far from full natural language understanding, though. But, at the same time, we’re so much father ahead than anyone seriously involved in AI in the late 2010’s could predict.

Let's turn our attention to language modeling, then. At the moment of writing, there is no doubt GPT-4 is the state of the art, at least in terms of being the most powerful, widely available general-purpose language model. That might not be the case in a couple of months when Gemini is finally out, but still, looking over at the story of the GPT series seems like the best way to understand how we got here and where we might go next.

How to train your chatbot

The history of chatbots goes all the way back to ELIZA in 1962, but their most recent incarnation started, at least in the public mind, with the advent of GPT-2 in 2019. Its ability to fabricate stories about talking unicorns found in the Peruvian mountains, a significant leap from anything we'd seen before, blew our minds. Sure, we had made strides in translation and text classification with bare transformers, but nothing came close to GPT-2's coherence in generating general-purpose text. Suddenly, an AI model could write a short story!

This was also the first time researchers decided not to immediately release the weights of the model, apparently not for any commercial reasons but for fear of it being misused to generate fake news and impersonate people online. Oh, the irony!

Next year, GPT-3 shook things up with its paper on large language models as few-shot learners, marking the first significant breakthrough in transfer learning. Transfer learning is a pivotal area in AI, exploring how quickly we can adapt to new tasks similar to ones we've already learned without starting from scratch. This was the focus of the GPT-3 paper, and the first time we had a hint that such a thing as general-purpose language models could exist.

The GPT-3 paper reveals something many linguists already believed: language understanding is a general-purpose task. Furthermore, text completion is also a very general-purpose task. If you excel at completing sentences across various domains, you also excel at a wide range of tasks that aren’t trivially just text completion, as many tasks can essentially be boiled down to “please complete the following text.” Summarizing text, translating to another language, or solving math problems are all examples of tasks easily framed as text completion.

However, there's a catch. While models like GPT-3 excel at predicting and completing text, they struggle to follow specific instructions. This happens because the training data doesn't resemble instructions. So, even though the model may know how to summarize or translate, you have to figure out how these tasks are presented in the training data to prompt the model effectively. For example, to summarize, you would give the model your text and append “TL-DR;.”

Learning to follow instructions

What if you want to give clear instructions to the model that allow for some flexibility in how you provide them? This is crucial because multiple ways exist to fill in a given text. For instance, when you ask GPT-3 a question, the text can be completed by providing an answer. Still, it is also valid to continue with similar questions, as vast collections of questions are available on the internet. The model doesn’t know whether you want more questions or the answer to the previous question.

That's where instruction fine-tuning comes in, as described in the InstructGPT paper. This approach enables you to train your model to better understand and respond to your instructions, whether asking it to solve an equation, summarize a text, generate ideas, or perform other tasks.

With instruction fine-tuning, the model can be guided to prioritize answering questions over simply executing completion. This approach functions by adjusting the likelihood of responses to questions. For this method to be effective, the model must possess a probable and correct answer to the question; we are just nudging it to value that answer more likely than other non-answer completions.

However, this is still not enough. For widespread use, you need a model willing to reject problematic instructions, friendly, and committed to providing factual and accurate information. The desired responses should align with user preferences and maintain a respectful and non-discriminatory tone, using generally informal yet polite language. In other words, you need a model that is knowledgeable (pretrained in the whole internet), helpful (can follow instructions), and nice. This is where reinforcement learning with human feedback (RLHF) comes into play, also presented in this paper.

Don’t be evil… please?

In RLHF, we take an instruction fine-tuned model and further adjust it to score better answers that align with user preferences —such as being more or less concise, using more or less formal language, or choosing to answer specific topics. This cannot be accomplished with supervised training because the range of possible human preferences is extensive and challenging to model in a dataset.

Researchers tackled this problem by turning to reinforcement learning. In this approach, we allow the model to provide multiple responses to a prompt rather than showing the model examples of good answers. A human evaluator then ranks and scores these responses based on their preferences for factuality, relevance, length and depth, language style, non-discrimination, etc. This ranking process encodes the human values we desire in a model but cannot program explicitly.

RLHF thus involves teaching the language model to rank its answers based on human preferences. The ranking model is then used to further train the language model, resulting in answers more aligned with user preferences.

From an engineering standpoint, this final puzzle piece transformed language models into usable applications. Just like, we had all the components leading to the iPhone, but the genius move was to put them together in a commercially viable and user-friendly format. ChatGPT is the iPhone of the AI revolution because it embodies several innovations in a commercially plausible manner, making it accessible and valuable to all users, not just experts or researchers.

And that’s the very brief story of how we got to ChatGPT. But that’s only the beginning. In the last year since ChatGPT was released, a new industry has grown around large language models. Let’s review some of the most interesting trends that make 2023 the year of artificial intelligence.

The Year of AI

Following ChatGPT, models have significantly diversified, with big players like Google, Meta, Apple, Twitter/X, Amazon, and many more training their large models. This competition has driven innovation: AI is now where the Big Tech giants fight each other.

Various open-source models have also emerged, stemming from Facebook Llama 1 and spinning off into a full Llamaverse. This has led to developing more specialized models, fine-tuned for specific domains and purposes, such as code generation, question-answering from databases, and more.

The open-source and closed-source models offer a wide range of trade-offs, and projects and wrappers are in place to seamlessly switch between these models, allowing for greater transparency and flexibility. This mirrors the trend in open source of creating diverse infrastructure and tools, as well as the emergence of frameworks that support various technologies. Just as we now have numerous options for databases, programming languages, and web frameworks, the field of language models is experiencing a similar surge in diversity and innovation.

The second major trend is multimodality. Although we focus on language models here, we must recognize the incredible development in visual and audio models. These models can recognize and analyze images, generate images and videos, and create audio outputs based on text prompts. Combining these separate models into multidomain models can solve problems requiring multiple modalities simultaneously. For instance, giving it a picture, asking questions about the image, modifying it, and generating new images based on it.

A third major trend is the rise of language models as application platforms. This means interfacing LLMs with plugins and applications and connecting with external code. ChatGPT introduced the idea of plugins that let you interact with external services such as knowledge bases, search engines, or APIs for specific products like a store website or booking service.

This concept has since evolved into language chains, where you can connect a linguistic interface with non-linguistic tools or services. This involves using a large language model to link with databases, retrieve relevant information, call APIs, perform actions, and compute results. With this setup, you can build applications, known as agents, that are language-driven and capable of interacting with the Internet to carry out tasks on your behalf. These agents go beyond just processing language; they can manage finances, build websites, and more, functioning closer to real assistants.

A final trend I want to highlight is the emergence of a new field: prompt engineering. It started as an informal practice due to the need for writing effective prompts, but has since evolved into a semi-formal discipline. Since these models aren't perfect and don't understand everything perfectly yet, the way you ask for information can affect the results you get. As a result, folks are working on figuring out best practices for crafting good prompts as these models continue to evolve.

Some experts are skeptical about the long-term future of prompt engineering, believing that as models get better at understanding, it won't matter how you prompt them. Others think prompt engineering will always be necessary, in the same sense that design patterns, naming and style conventions, and other programming practices remain relevant regardless of how powerful programming languages get.

The road ahead

Alright, we began with a quick overview of how we got here and where we stand. We've highlighted the potential capabilities these things can have. Now, let's delve into the challenges we've encountered over the past year that have proven difficult to overcome and what we can expect to solve in the near future.

Hallucinations

The first major challenge is the issue of hallucinations. Despite the models’ remarkable ability to produce coherent text aligned with user preferences, they can still veer off course into a response sequence that strays from the conversation. This randomness in sampling makes it impossible to guarantee that a model won't generate text that deviates from what's expected, no matter how finely tuned it gets. These deviations range from factual inaccuracies to more complex issues like promoting racism or discrimination. This phenomenon, often called “jail-breaking mode,” poses a significant obstacle to the model's reliability.

Addressing hallucinations becomes even more critical when using these models for tasks beyond writing or editing text. For instance, the model is used for tasks like booking flights or hotels, where the user may not have full visibility into the prompt used to generate the model's response, making it difficult to detect any hallucinations in the model's output.

These hallucinations are the primary roadblock to scaling language models for practical applications beyond simple conversational agents, presenting the most pressing issue for the widespread use of generative AI and large language models today.

Biases

Dealing with biases is another major challenge in making these models work in real-world applications. These models are trained on huge amounts of data, and as a result, they inevitably contain discriminatory and harmful biases inherited from the training data. When combined with the issues of hallucinations and jailbreak, it's always possible, whether intentionally or unintentionally, for these models to exhibit biased behavior.

Furthermore, as we use reinforcement learning and human feedback to encourage these models to behave fairly and unbiasedly, they lose their original effectiveness. They may refuse to answer slightly biased questions, some dubbed the “wokeness problem of AI.” That is, striving to make them as harmless and unbiased as possible can make them less responsive in situations where bias or discrimination isn't a concern.

Determining the precise boundary between a biased or discriminatory response and one that isn't is challenging when using reinforcement learning, which has a low sample density. As a result, the boundary of these two sets is likely to be quite jagged and hard to define. Consequently, we currently don't have a solution to unbias these models effectively. We don't yet know how to remove biases from a dataset or model without significantly harming performance.

This issue will continue to manifest in any situation where these models are employed to make or assist with decisions that involve people and ethical considerations regarding their data. Thus, as AI regulation becomes more prevalent, ensuring fairness is a major challenge in the widespread adoption of LLM-driven applications.

Understanding

Finally, let’s consider a more fundamental limitation: whether language models can develop accurate internal world models solely from linguistic interaction.

The GPT-4 paper, along with many anecdotal claims, suggests that sufficiently complex language models do form internal world models, enabling them to engage in something resembling actual general-purpose reasoning during next token prediction. However, skeptics from various fields question the ability to learn to build a world model purely from linguistic interactions, arguing that grounding in experiential reality is necessary.

An internal world model refers to an internal representation of the domain in which the language or machine learning model operates, allowing it to make predictions and plans. This is the first step to truly understanding some task or domain.

Long-term planning is a fundamental problem in AI that must be addressed for agents to effectively interact with the world and make decisions. The notion that accurate world models can be developed purely from language interactions suggests that these models could learn how processes work and perform inference, all from learning about the world through language alone.

Many researchers claim that large language models do not have internal representations. When prompted with complex sequences of instructions, these models can miss the intended meaning altogether. In contrast, humans can simulate these instructions in their minds, keeping track of the final outcome without remembering the entire sequence. This ability allows humans to accurately answer questions about the scenario in real-time.

An argument against LLMs’ internal models is that their error rates seem to increase with the length of prompts. As an example to understand this argument, consider a game of chess. In the game played mentally, one only needs to track the current board state to answer questions, regardless of the conversation's length. So, unless memory is faulty or one gets tired, there is no reason why the probability of making a mistake (like forgetting how a piece moves) would increase as the game gets longer if one has an internal model of the game.

However, without internal world representations, LLMs might only learn correlations between sequences of instructions and end states, leading to a higher likelihood of mistakes with longer prompts. This limitation is significant because if these language models are to be used for tasks beyond linguistic manipulation, such as planning, problem-solving, or decision-making, they must be able to simulate and predict outcomes for extended sequences. Without the ability to learn a model of the world, these models will be severely limited in the complexity of problems they can solve.

Furthermore, building internal models from linguistic interaction alone may be fundamentally impossible. If true, then we will need a qualitative improvement in our AI systems, a new approach that surpasses the capabilities of large language models.

Moving forward

As we say goodbye to 2023 and welcome 2024 into our lives, we can contemplate what has undoubtedly been the most impactful year of artificial intelligence. The next few years will be about cristalizing the many potential applications of AI into actual, useful products that might usher a new era of abundance like we’ve never seen before. Even if we hit a major theoretical barrier, and there is now new scientific development that has led to AGI for decades, we still have years ahead to harvest what’s already possible with current AI technology.

But the future is never certain. Artificial intelligence is the most powerful technology we have, and as such, it is also the most dangerous. There are deeply troubling problems and challenges in the short and medium term, for which we have little guarantee we can solve them. Only with the collective effort of researchers, engineers, lawmakers, creators, and the general public can we stand a chance to overcome them.

In any case, if you feel like the year has passed and you got left behind, worry not; 2024 is the best moment to get into AI. Whatever your profession and your interests, there is something in the field of AI for you. If you care about fundamental theory, AI questions are some of the most challenging open questions in math and logic. If you care about engineering, some of the most interesting pragmatic problems are about scaling AI. If you care about ethics and society, one of the most critical questions ahead is how to deal with this tech's massive impact on our lives. And if you just want to have fun, you now have one of the most powerful technologies ever made at your fingertips.

This is the last issue of Mostly Harmless until next year. I wanted to end on a high note because my articles are usually far more skeptical and cynical. However, I believe artificial intelligence has tremendous potential, and I’m dedicating my life to harness that potential. In 2024, I’ll be doubling down on these long-form articles, exploring the ins and outs of computer science and AI.

Hope to keep sharing them with you in 2024!