An excellently refreshing post. Based on your argument, it appears that, because LLMs needs scholastic methods to randomly select a token from the model vocabulary, it cannot come out of probabilistic token generation at all and that hinders the ability to truly reason. But if that’s the case don’t we all humans are also bound by the same vocab limitations? We get fine tuned on demand as & when we encounter new words/ sentences on the fly to “expand” our thinking ability. So probably (no pun intended 😊) some architecture change that accommodates on the fly training loop with novel methods to tap relevant vocabulary from the memory does the trick ?
Thanks! So, in a sense, I think this is a kind of an epistemic fallacy, isn't it? The fact that a given model is somewhat an accurate model of reality (for example, stochastic language modelling is a somewhat accurate model of human language generation) does not give the model causal explanatory power. It just gives the model predictive power. Meaning that I cannot go and claim that because this model "works" (in the sense it produces plausible predictions), then what must be going on in reality is anything similar at all. There is no reason why our brains need to be subject to the same limitations of the models we use to approximate it. In fact, is in this difference in limitations that the limits of the model more clearly arise. (I'm not sure if I'm making myself intelligible at all, or if I'm totally missing your point, so excuse me and correct me if that's the case). So, in conclusion, the current limitations of LLMs don't tell us anything at all about cognitive limitations in the human brain. Now, to your second argument that some novel architecture that can be retrained on the fly would do the trick... well, maybe, maybe not entirely, but I do think that would be a step in the right direction. If it would be enough, I don't think anyone at this point can tell :)
I do believe that your conclusion is correct but I am not sure I buy your argument. Interpolations are not necessarily hallucinations. If we sample a small number of coordinates from a linear function without noise, and then use OLS regression to fit the training data, the model.will always output the correct y value given any x, even if that x is not in the training data. You are correct that if we train an LLM on true natural language statements then apply it out of sample.it often does not produce true statements, but this not automatically follow from the fact that it is interpolating.
>The reason this is problematic is simple: any probabilistic model of language that can generate new sentences that didn't exist in the training data—that is, that can generalize at all—has to be, by definition, a hallucination machine. Or, to use Rob Nelson’s much better terminology: a confabulation machine.
You're right, Steve! The problem is not interpolation per se, its interpolation in the space of continuous embeddings of natural language sentences, trained by minimization of perplexity. The stochastic language model is not a true representation of language--like George Box said, all models are wrong, but some are useful. LLMs are very useful for modeling plausibility, but not that much for logical soundness. I'm sorry if the article is not explicit enough in this respect: I'm not claiming computers in general or even neural networks in particular are incapable of ever achieving a human-like level of intelligence; I'm just claiming the stochastic language modeling paradigm alone is not enough to get there.
The argument that was always put forward by proponents was that the model might arrive at genuine common sense inference as a lind.of “trick” to compress the training data, ie to use induction to arrive at something very close to human deduction. I think empirically we can see now that this hasn't happened. But I don't think that it was a priori clear beyond any doubt that this would not be the case, before we had even conducted the experiment. Although I wouldn't have bet a trillion dollars on a positive result, I would have been open minded about the possibility that the most efficient way for the network to predict the next token would be to internalise a model of the real world.causal interactions that gave rise to natural.language utterances about that world. If there were an uncontentious mathematical proof that this would never work, I don't think the experiment would have gone ahead. I still don't see any reason why it can't happen in principle. As a simple thought experiment, imagine that by sheer fluke a chance training process arrives at a network that really does predict next token through genuine deduction. Can we absolutely rule.out this case as having zero.probability over all.possine training runs and hyperparameter settings?
Searle had already demonstrated how Syntax is insufficient for semantic. You can't ever reason by syntax. This is my variation of Searle's CRA:
======
You memorize a whole bunch of shapes. Then, you memorize the order the shapes are supposed to go in so that if you see a bunch of shapes in a certain order, you would “answer” by picking a bunch of shapes in another prescribed order. Now, did you just learn any meaning behind any language?
======
...Looks familiar, doesn't it. That's pattern matching. Now, what is a machine and what does one do? A machine is "an assemblage of parts that transmit forces, motion, and energy one to another" (other non-machine senses are employing poetic license). What happens when a machine moves a load? It's _matching_ one load inside itself to _another_. Doesn't matter if it's a catapult, doesn't matter if it's a microscopic transistor in a microprocessor... That's what they ALL DO.
What really needs to be done, is people need to be taught what the heck a machine is. They are confused, schools aren't helping, and when they go out in the "workforce" they do confused research, make up badly anthropomorphized terms like "machine learning" (well it actually goes all the way back to "artificial intelligence" but I'm not here to write a book) AND screw up entire fields of _everything_ https://davidhsing.substack.com/p/what-the-world-needs-isnt-artificial
Thanks for such a thought-provoking argument! I agree with a part of it, but I don't agree with all of it. Let me try to unpack my thoughts here. Sorry for the long response, but I want to make sure I cover all the points.
First, if we go to the discussion that these stochastic language models that we currently have do not produce a good representation of the semantics of language, because there is no grounding of the symbols for the semantics to appear, that I 100% agree. These models are trained only with superficial correlations of words, therefore they cannot understand the semantics of what it means in the real context of the life of a human being, what it means, for example, the see the color red or to smell an apple...all these problems of qualia, right?
So, I agreeNow, that's one thing and it's very easy to argue.
Another very different thing is to say that a machine, by definition, can only capture the syntax and will never be able to capture the semantics. This is, I think, the fundamental flaw in John Searle's argument from the Chinese Room, and I think it is incorrect to say that John Searle *demonstrated* that a machine can only capture the syntax (sorry if I'm missinterpreting your comment here, please correct me if I am).
So, one thing is to argue that, if you are only able to capture the syntax and not the semantics, then there is no true understanding. This is not difficult to argue, in fact, this is the very definition of understanding: comprehension is in semantics, not the syntax level. Therefore, any process that only captures the syntax and not the semantics, by definition, is not understanding at all. There is nothing extraordinarily deep in that assertion.
Now, another thing altogether is the claim that in the Chinese Room proves machines only capture the syntax. The CRE is not a demonstration of that, it's an upfront assertion than what is happening is just syntax manipulation. The most common counterargument against the CRE is precisely to say that even if it is true that individual components of the Room only capture syntactic aspects of the problem, *the system as a whole*, since it is able to produce semantically correct answers for any input, then it must be by definition capturing the semantics. That is, the system as a whole does understand Chinese. Now, I don't know if I buy this argument, but it is hard to counter at first hand without begging the question. If you claim upfront that neither any component nor the system as a whole is able to capture the semantics, then the burden of proof is on you. That is a claim, not a conclusion.
Look at it the following way, in your brain, no individual neuron captures the semantics of language. All independent neurons are extremely stupid. However, you, as a system, do understand the semantics of language. Therefore, there is some type of emergent understanding that occurs at the level of the system that cannot be reduced to the components of the system.
So, trying to argue that a machine, by definition of machine, will only be able to capture the syntax and never the semantics is a strong claim that shifts the burden of proof to those making the claim. Now you have to tell me why is the human brain not a machine? What is the difference between a human brain and a machine that gives the brain some additional property that machines cannot have by definition?
That is, the hypothesis of computationalism--that cognition is fundamentally mechanistic--is the simplest hypothesis. It is the hypothesis that does not need to posit any new entities. Everything in the Universe is a machine. Some machines are sufficiently complex to have intelligence, consciousness, reasoning, qualia, etc. Other machines are not. The human brain is a sufficiently complex machine, computers--as we have them implemented so far--are not.
But to propose that computationalism is false, you have to tell me that there is something in the Universe that transcends the mechanistic nature of the Turing machine. And therefore, you have to You have to argue the existence of new types of entities. You may be right, but that hypothesis is more complex, and by no means self-evident.
On my side, I believe in a weak variant of computationalism, in which everything that is cognition and intelligence is completely mechanistic, but not necessarily consciousness and qualia. I don't necessarily believe or I'm not convinced that if you reconstruct all the brain mechanics in a different substrate, you get consciousness. But I'm sure that if you reconstruct the mechanics of the brain in a different substrate, you get all the cognitive capacities of the brain. And if the understanding of the semantics of a domain is necessary for some of those tasks, then that machine will have reached a semantic understanding.
My reformulation of CRA (I call it "Symbol Manipulator thought experiment" or SM) takes care of the man-in-box POV issue. In SM, man IS "box." The question posed at the end of SM:
"Now, did you just learn any meaning behind any language?"
Is the indictment. Obviously there's no meaning behind that shuffling of loads.
There isn't, and isn't going to be, any exhaustive functional modeling, and thus any evaluation such as "stupid" or other quantitative/qualitative labeling doesn't stick. While we'll never have exhaustive modeling of an underdetermined entity such as a biological neuron, we have COMPLETE visibility into machine functions. This is just another way of saying "it's a functionalist argument and none of them work" (cf. section of TDS article linked above named "Functionalist objections")
Note re: lack of modeling- "The unstated implication in most descriptions of neural coding is that the activity of neural networks is presented to an ideal observer or reader within the brain, often described as “downstream structures” that have access to the optimal way to decode the signals. But the ways in which such structures actually process those signals is unknown, and is rarely explicitly hypothesised, even in simple models of neural network function." https://www.theguardian.com/science/2020/feb/27/why-your-brain-is-not-a-computer-neuroscience-neural-networks-consciousness
"What is the difference between a human brain and a machine that gives the brain some additional property that machines cannot have by definition?"
Living beings obviously aren't machines. Two aspects:
1. See dictionary definition (Merriam Webster). As previously mentioned, a machine is "an assemblage of parts that transmit forces, motion, and energy one to another" (other non-machine senses are employing poetic license). Living beings aren't that, they are integral wholes that are grown, not assemblages at least in the normal sense of how we use those terms. We can't "disassemble" a living being... What happens when you try? You don't get discrete "parts," you get what only look like parts except you've damaged and CUT the surrounding tissues to do it. We've lived far too long under a mechanistic conception to even pay attention to this simple fact.
2. People arguing "AI" (functionalists particularly guilty) forget another fact, which is that all machines are artifacts. To even have a machine, you have to design and build one (sounds like Captain Obvious but... most people ignore that outright). Living beings aren't designed because that's not how the process of evolution works unless someone is arguing for "(divine) intelligent design. (even that is arguable because it could be said that such 'design' is talking about something completely different than what the word normally means)" What does that in turn entail?
2A) There is no such thing as "machine volition." (I explained that in the TDS article)
2B) Relatedly, no machine does anything "by itself." That's just nonsense. That very concept involves "design without design" and "programming without programming." NNs have algorithms too, just as any other piece of machinery (I used a catapult as example in the TDS article)
"you have to tell me that there is something in the Universe that transcends the mechanistic nature of the Turing machine."
You've given me a lot of food for thought, and I really appreciate it! I'm learning so much.
So, to be clear, I don't claim functionalism is true. In fact, functionalism makes some pretty strong claims that are very hard to agree with. What happens to me is that all theories of mind are more or less equally vague and unconvincing, none of them make falsifiable predictions (or else, we wouldn't be arguing about it), and, at some point, these discussion start to become discussions about the meaning of definitions.
Now, don't get me wrong, definitions do matter, and arguing about what the implications of definitions entail has led to some of the most fruitful developments in science: the whole field of math is just arguing about definitions. But, also, at some level, arguing about definitions becomes self-defeating.
If you insist on claiming that there is something a machine cannot do because of some dictionary definition of what machines are, then I will gladly grant you that claim. I just don't think that's extremely useful because the universe doesn't care about how we choose to define our concepts. And I think we can learn little to nothing about the real world by arguing how one definition of "machine" makes said concept capable or incapable of something.
My article is, I hope, leaning towards the pragmatic side of the argument. I don't care (in this article) if there is some ontological argument for the impossibility of reasoning in machines in general. All I care about is that there are theoretical and engineering reasons why our current implementation of a "thinking" machine is extremely limited in practice. I'm interested in this because we will have, in the near future, stuff built under these principles used to make life-or-death decisions (I fear) and we need to be aware of the danger. I think in this respect we both agree :)
Whether there is a more profound reason to uncover here, well, that's a discussion we can continue to have, and I'm happy to indulge in philosophical discussions. So thank you for continuing the conversation, now I have to read some of your links to satiate my curiosity.
Great article!... I learned a lot. The core arguments really connect and I believe them all, generally speaking. But I have to ask the big picture question. If AI can't technically reason, does it matter? I don't foresee LLMs making life-or-death decisions. I *DO* see AI more generally making those decisions. But those other AIs are trained on appropriate data for their task, not on human conversational language.
Thanks, Bill, for the comment. If that ends up being the case, I would be very relieved! But sadly I'm seeing a large push from mainstream LLM providers to present these as general purpose decision making tools, and claiming that all hallucinations and mistakes can be reduced to zero by virtue of scaling.
So yeah, if (I hope so) we end up ditching LLMs as general-purpose reasoners and just use them for the cases they are actually useful for (which may include some instances of not formally verifiable reasoning), then this argument only matters for people like me who are studying LLMs from an academic perspective.
Yes, I definitely see your point. It's hard to be an optimist for AI these days because the space is already claimed by hypesters and evangelists. I hope I am not one.
For sure I don't believe that hallucinations ever go to zero in an LLM; nor that they're good for all decisions or even close. And yes, I see claims like that out there, unfortunately.
Your article was very thought-provoking in that it acknowledged, as many others don't acknowledge, the idea of other AI outside the scope of LLMs... and for that matter even non-AI programs outside of LLMs. An LLM can't do formal math; but I'm guessing it will learn (ne; someone will train it) to call on an SAT solver when needed. That kind of connection is the future, IMO.
My own background is autonomous systems and how to make them safe... and that kind of system has a very different profile of knowledge compared to LLMs.
I definitely believe integration is the way forward. LLMs are really good for one thing: as a linguistic interface. We don't need to bet everything on this one thing being the solution to general purpose AI. It most likely won't. It will be a collection of carefully engineered systems working together, as everything else.
" I don't foresee LLMs making life-or-death decisions. I *DO* see AI more generally making those decisions. But those other AIs are trained on appropriate data"
The problem is with neural networks in general, and the only "appropriate data" is the infinitely large set called "the real world" where anything and everything that aren't anticipated gets labeled as "edge case".
The point is technically correct. But also in my opinion, (sorry, not trying to be rude) largely irrelevant.
First, regarding Tesla, they irresponsibly deployed tech that was not ready. Every other automaker has the same kind of system in their labs; but all the others made the (correct) decision not to deploy it at that time, because of known hazardous scenarios like the one you mention. I deplore Tesla for their inaccurate and deceiving claims and I won't defend them. But despite their arrogance, and setting aside Tesla itself, the future of self-driving is very real and is becoming safer than human drivers as we speak. "Becoming," .. arguably not there yet. But if you look at more responsible actors (Waymo, Zoox, some others), the data is getting better than humans, at roughly the current time.
To paraphrase my own article: humans are not the deductive logic geniuses we're so often taught about. We ourselves are also big inductive inference machines (albeit with some deductive reasoning that sometimes excels AI. And sometimes not). Humans also struggle withe edge cases. And like NNs, we'll have to live with some mistakes from ourselves. The question evolves into this: given the imperfections, who makes more (and more severe) mistakes? Depends a lot on the detail...
I assert that lack of reasoning in AI is largely irrelevant for the real-world use of AI; and I include safety-critical applications in my scope of the assertion. As I am making a practical argument, I will provide practical evidence to back it up.
Below is a link to the ISO 8800 standard, released in December of last year. It addresses the topic of "Road Vehicles - Safety and Artificial Intelligence"; and is devoted largely to AI/NNs in semi-autonomous and fully-autonomous driving. The standard is 167 pages long and was written by roughly 100-200 industry experts (mostly AI, software, and safety experts). It contains various frameworks, requirements, and best practices for safe use of AI in vehicles.
It can be argued that NNs are bad or that AI doesn't reason. In both cases: maybe right, maybe wrong, maybe shades of grey. But those points are (as I asserted) basically irrelevant. If you drive past a new car dealership and look at the new cars, almost all of them contain NNs within them already.
You didn't raise the usual one that LLM's are just using abstract tokens so have no concept of reality. (Guess it doesn't matter) But an LLM can have direct experience of numbers (eg no of words in a sentence) so it ought to be able to work out number theory from first principles.
Personally I feel my speech handling works pretty much like an LLM but presumably I have connected accessory modules to enable more meaningful thoughts.
You're absolutely right. I didn't want to go as deep as symbol grounding for this argument because we don't need to... LLMs are already brittle as they stand, even by design. OTOH, I suppose maybe an unembodied intelligence could come up with number theory, no grounding needed for that, but still, that intelligence needs to be at least as powerful as a Turing machine. LLMs don't even make that cut.
You can catch CoPilot doing self-critique. I playfully asked it a few questions about human reproduction and race (not in the same question!). It typed a couple of lines, then thought better of it and wiped the screen before saying that it couldn't answer questions like that.
Well done. 👏 That is one of the best debunkings of the hype that I have seen. It blows the the smoke screens away. I agree we need an alternate model. LLMs are components not solutions. .
An excellently refreshing post. Based on your argument, it appears that, because LLMs needs scholastic methods to randomly select a token from the model vocabulary, it cannot come out of probabilistic token generation at all and that hinders the ability to truly reason. But if that’s the case don’t we all humans are also bound by the same vocab limitations? We get fine tuned on demand as & when we encounter new words/ sentences on the fly to “expand” our thinking ability. So probably (no pun intended 😊) some architecture change that accommodates on the fly training loop with novel methods to tap relevant vocabulary from the memory does the trick ?
Thanks! So, in a sense, I think this is a kind of an epistemic fallacy, isn't it? The fact that a given model is somewhat an accurate model of reality (for example, stochastic language modelling is a somewhat accurate model of human language generation) does not give the model causal explanatory power. It just gives the model predictive power. Meaning that I cannot go and claim that because this model "works" (in the sense it produces plausible predictions), then what must be going on in reality is anything similar at all. There is no reason why our brains need to be subject to the same limitations of the models we use to approximate it. In fact, is in this difference in limitations that the limits of the model more clearly arise. (I'm not sure if I'm making myself intelligible at all, or if I'm totally missing your point, so excuse me and correct me if that's the case). So, in conclusion, the current limitations of LLMs don't tell us anything at all about cognitive limitations in the human brain. Now, to your second argument that some novel architecture that can be retrained on the fly would do the trick... well, maybe, maybe not entirely, but I do think that would be a step in the right direction. If it would be enough, I don't think anyone at this point can tell :)
I do believe that your conclusion is correct but I am not sure I buy your argument. Interpolations are not necessarily hallucinations. If we sample a small number of coordinates from a linear function without noise, and then use OLS regression to fit the training data, the model.will always output the correct y value given any x, even if that x is not in the training data. You are correct that if we train an LLM on true natural language statements then apply it out of sample.it often does not produce true statements, but this not automatically follow from the fact that it is interpolating.
>The reason this is problematic is simple: any probabilistic model of language that can generate new sentences that didn't exist in the training data—that is, that can generalize at all—has to be, by definition, a hallucination machine. Or, to use Rob Nelson’s much better terminology: a confabulation machine.
You're right, Steve! The problem is not interpolation per se, its interpolation in the space of continuous embeddings of natural language sentences, trained by minimization of perplexity. The stochastic language model is not a true representation of language--like George Box said, all models are wrong, but some are useful. LLMs are very useful for modeling plausibility, but not that much for logical soundness. I'm sorry if the article is not explicit enough in this respect: I'm not claiming computers in general or even neural networks in particular are incapable of ever achieving a human-like level of intelligence; I'm just claiming the stochastic language modeling paradigm alone is not enough to get there.
The argument that was always put forward by proponents was that the model might arrive at genuine common sense inference as a lind.of “trick” to compress the training data, ie to use induction to arrive at something very close to human deduction. I think empirically we can see now that this hasn't happened. But I don't think that it was a priori clear beyond any doubt that this would not be the case, before we had even conducted the experiment. Although I wouldn't have bet a trillion dollars on a positive result, I would have been open minded about the possibility that the most efficient way for the network to predict the next token would be to internalise a model of the real world.causal interactions that gave rise to natural.language utterances about that world. If there were an uncontentious mathematical proof that this would never work, I don't think the experiment would have gone ahead. I still don't see any reason why it can't happen in principle. As a simple thought experiment, imagine that by sheer fluke a chance training process arrives at a network that really does predict next token through genuine deduction. Can we absolutely rule.out this case as having zero.probability over all.possine training runs and hyperparameter settings?
I go by a much simpler explanation. LLMs, and machines in general, can't refer to anything at all.
Linguistics POV1:
If entity E couldn't refer to a specific item X, then how could it reason _about_ X?
https://davidhsing.substack.com/p/why-neural-networks-is-a-bad-technology
Linguistics POV2:
Searle had already demonstrated how Syntax is insufficient for semantic. You can't ever reason by syntax. This is my variation of Searle's CRA:
======
You memorize a whole bunch of shapes. Then, you memorize the order the shapes are supposed to go in so that if you see a bunch of shapes in a certain order, you would “answer” by picking a bunch of shapes in another prescribed order. Now, did you just learn any meaning behind any language?
======
...Looks familiar, doesn't it. That's pattern matching. Now, what is a machine and what does one do? A machine is "an assemblage of parts that transmit forces, motion, and energy one to another" (other non-machine senses are employing poetic license). What happens when a machine moves a load? It's _matching_ one load inside itself to _another_. Doesn't matter if it's a catapult, doesn't matter if it's a microscopic transistor in a microprocessor... That's what they ALL DO.
What really needs to be done, is people need to be taught what the heck a machine is. They are confused, schools aren't helping, and when they go out in the "workforce" they do confused research, make up badly anthropomorphized terms like "machine learning" (well it actually goes all the way back to "artificial intelligence" but I'm not here to write a book) AND screw up entire fields of _everything_ https://davidhsing.substack.com/p/what-the-world-needs-isnt-artificial
Thanks for such a thought-provoking argument! I agree with a part of it, but I don't agree with all of it. Let me try to unpack my thoughts here. Sorry for the long response, but I want to make sure I cover all the points.
First, if we go to the discussion that these stochastic language models that we currently have do not produce a good representation of the semantics of language, because there is no grounding of the symbols for the semantics to appear, that I 100% agree. These models are trained only with superficial correlations of words, therefore they cannot understand the semantics of what it means in the real context of the life of a human being, what it means, for example, the see the color red or to smell an apple...all these problems of qualia, right?
So, I agreeNow, that's one thing and it's very easy to argue.
Another very different thing is to say that a machine, by definition, can only capture the syntax and will never be able to capture the semantics. This is, I think, the fundamental flaw in John Searle's argument from the Chinese Room, and I think it is incorrect to say that John Searle *demonstrated* that a machine can only capture the syntax (sorry if I'm missinterpreting your comment here, please correct me if I am).
So, one thing is to argue that, if you are only able to capture the syntax and not the semantics, then there is no true understanding. This is not difficult to argue, in fact, this is the very definition of understanding: comprehension is in semantics, not the syntax level. Therefore, any process that only captures the syntax and not the semantics, by definition, is not understanding at all. There is nothing extraordinarily deep in that assertion.
Now, another thing altogether is the claim that in the Chinese Room proves machines only capture the syntax. The CRE is not a demonstration of that, it's an upfront assertion than what is happening is just syntax manipulation. The most common counterargument against the CRE is precisely to say that even if it is true that individual components of the Room only capture syntactic aspects of the problem, *the system as a whole*, since it is able to produce semantically correct answers for any input, then it must be by definition capturing the semantics. That is, the system as a whole does understand Chinese. Now, I don't know if I buy this argument, but it is hard to counter at first hand without begging the question. If you claim upfront that neither any component nor the system as a whole is able to capture the semantics, then the burden of proof is on you. That is a claim, not a conclusion.
Look at it the following way, in your brain, no individual neuron captures the semantics of language. All independent neurons are extremely stupid. However, you, as a system, do understand the semantics of language. Therefore, there is some type of emergent understanding that occurs at the level of the system that cannot be reduced to the components of the system.
So, trying to argue that a machine, by definition of machine, will only be able to capture the syntax and never the semantics is a strong claim that shifts the burden of proof to those making the claim. Now you have to tell me why is the human brain not a machine? What is the difference between a human brain and a machine that gives the brain some additional property that machines cannot have by definition?
That is, the hypothesis of computationalism--that cognition is fundamentally mechanistic--is the simplest hypothesis. It is the hypothesis that does not need to posit any new entities. Everything in the Universe is a machine. Some machines are sufficiently complex to have intelligence, consciousness, reasoning, qualia, etc. Other machines are not. The human brain is a sufficiently complex machine, computers--as we have them implemented so far--are not.
But to propose that computationalism is false, you have to tell me that there is something in the Universe that transcends the mechanistic nature of the Turing machine. And therefore, you have to You have to argue the existence of new types of entities. You may be right, but that hypothesis is more complex, and by no means self-evident.
On my side, I believe in a weak variant of computationalism, in which everything that is cognition and intelligence is completely mechanistic, but not necessarily consciousness and qualia. I don't necessarily believe or I'm not convinced that if you reconstruct all the brain mechanics in a different substrate, you get consciousness. But I'm sure that if you reconstruct the mechanics of the brain in a different substrate, you get all the cognitive capacities of the brain. And if the understanding of the semantics of a domain is necessary for some of those tasks, then that machine will have reached a semantic understanding.
Hope it makes some sense :)
My reformulation of CRA (I call it "Symbol Manipulator thought experiment" or SM) takes care of the man-in-box POV issue. In SM, man IS "box." The question posed at the end of SM:
"Now, did you just learn any meaning behind any language?"
Is the indictment. Obviously there's no meaning behind that shuffling of loads.
The concept is the same... Searle's CRA's defect is only rhetorical in nature as I've pointed out in this longer explanation: https://towardsdatascience.com/artificial-consciousness-is-impossible-c1b2ab0bdc46/ (section "Symbol Manipulator, a thought experiment")
"All independent neurons are extremely stupid."
There isn't, and isn't going to be, any exhaustive functional modeling, and thus any evaluation such as "stupid" or other quantitative/qualitative labeling doesn't stick. While we'll never have exhaustive modeling of an underdetermined entity such as a biological neuron, we have COMPLETE visibility into machine functions. This is just another way of saying "it's a functionalist argument and none of them work" (cf. section of TDS article linked above named "Functionalist objections")
Note re: lack of modeling- "The unstated implication in most descriptions of neural coding is that the activity of neural networks is presented to an ideal observer or reader within the brain, often described as “downstream structures” that have access to the optimal way to decode the signals. But the ways in which such structures actually process those signals is unknown, and is rarely explicitly hypothesised, even in simple models of neural network function." https://www.theguardian.com/science/2020/feb/27/why-your-brain-is-not-a-computer-neuroscience-neural-networks-consciousness
"What is the difference between a human brain and a machine that gives the brain some additional property that machines cannot have by definition?"
Living beings obviously aren't machines. Two aspects:
1. See dictionary definition (Merriam Webster). As previously mentioned, a machine is "an assemblage of parts that transmit forces, motion, and energy one to another" (other non-machine senses are employing poetic license). Living beings aren't that, they are integral wholes that are grown, not assemblages at least in the normal sense of how we use those terms. We can't "disassemble" a living being... What happens when you try? You don't get discrete "parts," you get what only look like parts except you've damaged and CUT the surrounding tissues to do it. We've lived far too long under a mechanistic conception to even pay attention to this simple fact.
2. People arguing "AI" (functionalists particularly guilty) forget another fact, which is that all machines are artifacts. To even have a machine, you have to design and build one (sounds like Captain Obvious but... most people ignore that outright). Living beings aren't designed because that's not how the process of evolution works unless someone is arguing for "(divine) intelligent design. (even that is arguable because it could be said that such 'design' is talking about something completely different than what the word normally means)" What does that in turn entail?
2A) There is no such thing as "machine volition." (I explained that in the TDS article)
2B) Relatedly, no machine does anything "by itself." That's just nonsense. That very concept involves "design without design" and "programming without programming." NNs have algorithms too, just as any other piece of machinery (I used a catapult as example in the TDS article)
"you have to tell me that there is something in the Universe that transcends the mechanistic nature of the Turing machine."
Not everything deals with information. Reference the famous "Mary in the monochrome room" thought experiment https://plato.stanford.edu/entries/qualia-knowledge/
"if you reconstruct all the brain mechanics in a different substrate"
Functionalist arguments all fail. (cf. section of TDS article named "Functionalist objections" https://towardsdatascience.com/artificial-consciousness-is-impossible-c1b2ab0bdc46/)
You've given me a lot of food for thought, and I really appreciate it! I'm learning so much.
So, to be clear, I don't claim functionalism is true. In fact, functionalism makes some pretty strong claims that are very hard to agree with. What happens to me is that all theories of mind are more or less equally vague and unconvincing, none of them make falsifiable predictions (or else, we wouldn't be arguing about it), and, at some point, these discussion start to become discussions about the meaning of definitions.
Now, don't get me wrong, definitions do matter, and arguing about what the implications of definitions entail has led to some of the most fruitful developments in science: the whole field of math is just arguing about definitions. But, also, at some level, arguing about definitions becomes self-defeating.
If you insist on claiming that there is something a machine cannot do because of some dictionary definition of what machines are, then I will gladly grant you that claim. I just don't think that's extremely useful because the universe doesn't care about how we choose to define our concepts. And I think we can learn little to nothing about the real world by arguing how one definition of "machine" makes said concept capable or incapable of something.
My article is, I hope, leaning towards the pragmatic side of the argument. I don't care (in this article) if there is some ontological argument for the impossibility of reasoning in machines in general. All I care about is that there are theoretical and engineering reasons why our current implementation of a "thinking" machine is extremely limited in practice. I'm interested in this because we will have, in the near future, stuff built under these principles used to make life-or-death decisions (I fear) and we need to be aware of the danger. I think in this respect we both agree :)
Whether there is a more profound reason to uncover here, well, that's a discussion we can continue to have, and I'm happy to indulge in philosophical discussions. So thank you for continuing the conversation, now I have to read some of your links to satiate my curiosity.
Some sections of TDS are timing out. No idea why. Here's a copy on Medium https://medium.com/towards-data-science/artificial-consciousness-is-impossible-c1b2ab0bdc46
Great article!... I learned a lot. The core arguments really connect and I believe them all, generally speaking. But I have to ask the big picture question. If AI can't technically reason, does it matter? I don't foresee LLMs making life-or-death decisions. I *DO* see AI more generally making those decisions. But those other AIs are trained on appropriate data for their task, not on human conversational language.
In the broader sense, while I agree LLMs don't reason, I believe the fact is largely irrelevant. More more contained in this article (just published literally today); I would be very interested in your comment: https://billatsystematica.substack.com/p/will-ai-obtain-human-characteristics
Thanks, Bill, for the comment. If that ends up being the case, I would be very relieved! But sadly I'm seeing a large push from mainstream LLM providers to present these as general purpose decision making tools, and claiming that all hallucinations and mistakes can be reduced to zero by virtue of scaling.
So yeah, if (I hope so) we end up ditching LLMs as general-purpose reasoners and just use them for the cases they are actually useful for (which may include some instances of not formally verifiable reasoning), then this argument only matters for people like me who are studying LLMs from an academic perspective.
Yes, I definitely see your point. It's hard to be an optimist for AI these days because the space is already claimed by hypesters and evangelists. I hope I am not one.
For sure I don't believe that hallucinations ever go to zero in an LLM; nor that they're good for all decisions or even close. And yes, I see claims like that out there, unfortunately.
Your article was very thought-provoking in that it acknowledged, as many others don't acknowledge, the idea of other AI outside the scope of LLMs... and for that matter even non-AI programs outside of LLMs. An LLM can't do formal math; but I'm guessing it will learn (ne; someone will train it) to call on an SAT solver when needed. That kind of connection is the future, IMO.
My own background is autonomous systems and how to make them safe... and that kind of system has a very different profile of knowledge compared to LLMs.
I definitely believe integration is the way forward. LLMs are really good for one thing: as a linguistic interface. We don't need to bet everything on this one thing being the solution to general purpose AI. It most likely won't. It will be a collection of carefully engineered systems working together, as everything else.
" I don't foresee LLMs making life-or-death decisions. I *DO* see AI more generally making those decisions. But those other AIs are trained on appropriate data"
The problem is with neural networks in general, and the only "appropriate data" is the infinitely large set called "the real world" where anything and everything that aren't anticipated gets labeled as "edge case".
https://davidhsing.substack.com/p/why-neural-networks-is-a-bad-technology
The point is technically correct. But also in my opinion, (sorry, not trying to be rude) largely irrelevant.
First, regarding Tesla, they irresponsibly deployed tech that was not ready. Every other automaker has the same kind of system in their labs; but all the others made the (correct) decision not to deploy it at that time, because of known hazardous scenarios like the one you mention. I deplore Tesla for their inaccurate and deceiving claims and I won't defend them. But despite their arrogance, and setting aside Tesla itself, the future of self-driving is very real and is becoming safer than human drivers as we speak. "Becoming," .. arguably not there yet. But if you look at more responsible actors (Waymo, Zoox, some others), the data is getting better than humans, at roughly the current time.
To paraphrase my own article: humans are not the deductive logic geniuses we're so often taught about. We ourselves are also big inductive inference machines (albeit with some deductive reasoning that sometimes excels AI. And sometimes not). Humans also struggle withe edge cases. And like NNs, we'll have to live with some mistakes from ourselves. The question evolves into this: given the imperfections, who makes more (and more severe) mistakes? Depends a lot on the detail...
You're arguing by assertion. I gave something to back myself up, what about you?
I assert that lack of reasoning in AI is largely irrelevant for the real-world use of AI; and I include safety-critical applications in my scope of the assertion. As I am making a practical argument, I will provide practical evidence to back it up.
Below is a link to the ISO 8800 standard, released in December of last year. It addresses the topic of "Road Vehicles - Safety and Artificial Intelligence"; and is devoted largely to AI/NNs in semi-autonomous and fully-autonomous driving. The standard is 167 pages long and was written by roughly 100-200 industry experts (mostly AI, software, and safety experts). It contains various frameworks, requirements, and best practices for safe use of AI in vehicles.
It can be argued that NNs are bad or that AI doesn't reason. In both cases: maybe right, maybe wrong, maybe shades of grey. But those points are (as I asserted) basically irrelevant. If you drive past a new car dealership and look at the new cars, almost all of them contain NNs within them already.
https://webstore.ansi.org/standards/iso/isopas88002024?gad_source=1&gclid=CjwKCAiA5pq-BhBuEiwAvkzVZSoDWM5nrdLR5GdkhschU0rBeayWqrekjGE7aEWyaG-aBCI-j98_3RoCQIkQAvD_BwE
It goes without saying that "There is a safety standard for X, therefore X is safe" is a fallacious line of reasoning.
That's not any kind of evidence.
What I have shown on my linked Substack post is:
"X is evidently unsafe, thereful X is unsafe."
Please try again.
I don't believe I said X is safe. Did I? Please cite.
Extraordinarily helpful, thanks.
Excellent post.
You didn't raise the usual one that LLM's are just using abstract tokens so have no concept of reality. (Guess it doesn't matter) But an LLM can have direct experience of numbers (eg no of words in a sentence) so it ought to be able to work out number theory from first principles.
Personally I feel my speech handling works pretty much like an LLM but presumably I have connected accessory modules to enable more meaningful thoughts.
You're absolutely right. I didn't want to go as deep as symbol grounding for this argument because we don't need to... LLMs are already brittle as they stand, even by design. OTOH, I suppose maybe an unembodied intelligence could come up with number theory, no grounding needed for that, but still, that intelligence needs to be at least as powerful as a Turing machine. LLMs don't even make that cut.
You can catch CoPilot doing self-critique. I playfully asked it a few questions about human reproduction and race (not in the same question!). It typed a couple of lines, then thought better of it and wiped the screen before saying that it couldn't answer questions like that.
Oh yeah they have output filters, it's a neat hack but not sufficient but reliable self-critique.
yes, you could hack it with a screen recorder. :)
Since I wrote the above it dawned on me that LLM's are basically Meaning Zombies, and demonstrate why our evolution didn't stop at that point.
Well done. 👏 That is one of the best debunkings of the hype that I have seen. It blows the the smoke screens away. I agree we need an alternate model. LLMs are components not solutions. .