Fantastic description of LLMs. I've always said they were intended to be linguistically accurate, not factually accurate but I haven't thought that the hallucinations are a feature, not a bug.
Thank you for writing, Alejandro. I hope you are doing well. It's very likely that as more powerful LLMs emerge and are fine-tuned, we will build huge vector databases and knowledge graphs of factual information (or relational factual information) that will be able to heuristically or epistemologically judge the factuality of an LLM answer in O(1), based on truth. This could, I imagine (remember, I recently started this journey into ML/AI), be done at different stages (embedding vs. "output" vs. training vs. prompt) of the entire pipeline. Thus, while the hallucination problem is inherent in the way transformer architectures work today, there may be holistic approaches that include transformer models but rely on other methods and strategies to generate the final output, which can significantly reduce the likelihood of hallucinations for otherwise factual knowledge by grounding the model at different stages of the development and deployment process. What do you think? cheers.
Very thoughtful piece today! I've been thinking about this phenomenon for quite some time now, really ever since I noticed that there was a slider bar for one of the LLMs. On one side was something like "factual accuracy" and the other side was "creativity."
I started thinking about how our own minds work, and that's about as far as I've gotten (turns out I'm in great company, since literally nobody knows how our minds work).
I love how you distinguish between Hallucinations, out of distribution erros, and bias.
I think for a given prompt, you could also map out the distribution of hallucinations, and may find that it's actually fat tailed? That would be interesting.
The other thing I always think about is "hallucination inheritance". Once an LLM has hallucinated something, it might be feeding into another LLM which would inherit that hallucination, add more, or modify it. It eventually becomes an untraceable game of chinese whispers.
I can see this happening in electronic health records, where lots of ambient AI tools are already out there summarising doctor-patient conversations.
Thanks for this exploration. You're not the first one to point out that hallucinations aren't "solvable" within the current LLM architecture.
I find it fascinating that AI is kind of caught in this awkward middle of being subpar for any of its potential applications:
If you're using AI for practical, data-grounded purposes, you have to contend with hallucinations and unreliability.
If you want to use it for stuff where facts don't matter, like creative writing, you run into the issue of recycled prose and themes.
I feel like one of the most useful applications in my own life is using LLMs to brainstorm - LLMs are often "creative" enough to nudge me into an interesting direction while being able to output lots of ideas very quickly.
Apr 24·edited Apr 24Liked by Alejandro Piad Morffis
What reliable AI requires is a chain of ethical responsibility and keeping fingers off the scales.
Black box systems are a problem, not a solution. Reality is biased and we need to let AI have access to that bias before it can understand how help us. Censorship and puritanical rules only harm the ability of the system to produce meaningful answers.
Great article, Alejandro! Your knowledge on these topics is obvious, but so is your skill in teaching. Thanks for writing another great article that takes complex (but important to understand) subjects and makes them accessible.
I appreciate your nod to expectation setting/user training. As much as any of this is a technical problem, I think there are just as many product challenges around how we enable users to interact with the underlying technology.
Apr 12·edited Apr 12Liked by Alejandro Piad Morffis
"The reason this mainly works is that generating plausibly sounding text has a high probability of reproducing something that is true, provided you are trained on mostly truthful data. However, large language models (LLMs) are trained on vast corpora of text data from the internet, which contains inaccuracies, biases, and even fabricated information". Thank you for this clear and enlightening issue! I also recommend this interesting article from Scientific American: https://www.scientificamerican.com/article/chatbot-hallucinations-inevitable/#:~:text=The%20real%20problem%2C%20according%20to,that%20leave%20no%20chatbot%20unsupervised, according to which “AI Chatbots Will Never Stop Hallucinating”.
It is not quite the same issue as bias, but it is at the core of the matter of truth— because truth relies on the dependability of language for its expression. There are an increasing number of words that now have inverted meanings (including the word‘truth’) that we laughingly write off as oxymorons. But LLMs lack the subtlety to adjust for them. It’s bad news for AI but good news for humans perhaps!
A very lucid piece. Thanks for that. I have a real concern relating to semantics and how machine learning can cope with the deliberate alteration of meaning, as for instance in the Orwellian approach taken in Critical Theory. How do you think that can be factored?
Fantastic description of LLMs. I've always said they were intended to be linguistically accurate, not factually accurate but I haven't thought that the hallucinations are a feature, not a bug.
Thank you for writing, Alejandro. I hope you are doing well. It's very likely that as more powerful LLMs emerge and are fine-tuned, we will build huge vector databases and knowledge graphs of factual information (or relational factual information) that will be able to heuristically or epistemologically judge the factuality of an LLM answer in O(1), based on truth. This could, I imagine (remember, I recently started this journey into ML/AI), be done at different stages (embedding vs. "output" vs. training vs. prompt) of the entire pipeline. Thus, while the hallucination problem is inherent in the way transformer architectures work today, there may be holistic approaches that include transformer models but rely on other methods and strategies to generate the final output, which can significantly reduce the likelihood of hallucinations for otherwise factual knowledge by grounding the model at different stages of the development and deployment process. What do you think? cheers.
Very thoughtful piece today! I've been thinking about this phenomenon for quite some time now, really ever since I noticed that there was a slider bar for one of the LLMs. On one side was something like "factual accuracy" and the other side was "creativity."
I started thinking about how our own minds work, and that's about as far as I've gotten (turns out I'm in great company, since literally nobody knows how our minds work).
I love how you distinguish between Hallucinations, out of distribution erros, and bias.
I think for a given prompt, you could also map out the distribution of hallucinations, and may find that it's actually fat tailed? That would be interesting.
The other thing I always think about is "hallucination inheritance". Once an LLM has hallucinated something, it might be feeding into another LLM which would inherit that hallucination, add more, or modify it. It eventually becomes an untraceable game of chinese whispers.
I can see this happening in electronic health records, where lots of ambient AI tools are already out there summarising doctor-patient conversations.
What are your thoughts?
This article is at least 78% factual!
Thanks for this exploration. You're not the first one to point out that hallucinations aren't "solvable" within the current LLM architecture.
I find it fascinating that AI is kind of caught in this awkward middle of being subpar for any of its potential applications:
If you're using AI for practical, data-grounded purposes, you have to contend with hallucinations and unreliability.
If you want to use it for stuff where facts don't matter, like creative writing, you run into the issue of recycled prose and themes.
I feel like one of the most useful applications in my own life is using LLMs to brainstorm - LLMs are often "creative" enough to nudge me into an interesting direction while being able to output lots of ideas very quickly.
What reliable AI requires is a chain of ethical responsibility and keeping fingers off the scales.
Black box systems are a problem, not a solution. Reality is biased and we need to let AI have access to that bias before it can understand how help us. Censorship and puritanical rules only harm the ability of the system to produce meaningful answers.
are LLM 'hallucinations' a result of the models themselves or the illogical nature of the questions they're asked? your post and others inspired a poem: https://open.substack.com/pub/cybilxtheais/p/matchstick-dissonance?r=2ar57s&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true
Great article, Alejandro! Your knowledge on these topics is obvious, but so is your skill in teaching. Thanks for writing another great article that takes complex (but important to understand) subjects and makes them accessible.
I appreciate your nod to expectation setting/user training. As much as any of this is a technical problem, I think there are just as many product challenges around how we enable users to interact with the underlying technology.
"The reason this mainly works is that generating plausibly sounding text has a high probability of reproducing something that is true, provided you are trained on mostly truthful data. However, large language models (LLMs) are trained on vast corpora of text data from the internet, which contains inaccuracies, biases, and even fabricated information". Thank you for this clear and enlightening issue! I also recommend this interesting article from Scientific American: https://www.scientificamerican.com/article/chatbot-hallucinations-inevitable/#:~:text=The%20real%20problem%2C%20according%20to,that%20leave%20no%20chatbot%20unsupervised, according to which “AI Chatbots Will Never Stop Hallucinating”.
It is not quite the same issue as bias, but it is at the core of the matter of truth— because truth relies on the dependability of language for its expression. There are an increasing number of words that now have inverted meanings (including the word‘truth’) that we laughingly write off as oxymorons. But LLMs lack the subtlety to adjust for them. It’s bad news for AI but good news for humans perhaps!
A very lucid piece. Thanks for that. I have a real concern relating to semantics and how machine learning can cope with the deliberate alteration of meaning, as for instance in the Orwellian approach taken in Critical Theory. How do you think that can be factored?