A colleague posted a really good video about the reality of how AI works, and what happens when AI gets facts wrong:
Quick summary in case you don’t want to watch it: he describes the process by which AI arrives at answers to prompts, and why agents are better at things like, say, poetry versus straight-up mathematical equations. (He’s got great analogies, btw.)
It's a very good video, but he kind of drops the ball on one of the key elements to why Poetry works and the math does not, and it has nothing to do with the machine or the programming or the prediction. It has to do with the fact that AI does not exist in a vacuum. The entire premise of prediction is that it is predicting what the audience expects.
Poetry - even as a literary form - lives in the gaps of language and communication. Poetry is symbolic, which means that people actively insert meaning between the words. We do this with prose, too, but with poetry there are far more gaps to fall into.
"... and miles to go before I sleep" works not because it's a shared metaphor, but because it is a contributed shared metaphor - contributed by the interpretation of the user. It's dependent upon the English semantic comparison of "death" and "sleeping," which may or may not exist in other languages. It's one of the reasons why the Odyssey is so damn hard to correctly translate into other cultures, for instance.
This is why we hear stories of people "falling in love" with their AI agents, but it's no different than the anecdotes of people imagining real relationships with soap opera stars in the 70s and 80s. The very act of 'trusting' AI is one of audience investment, not AI integrity.
In Communication Theory, we use the Shannon-Weaver model as the foundation for nearly every aspect of interaction - both with human/human and human/computer. The model - at its core - is very simple:
Source - Message - Medium - Noise - Receiver - Feedback
In the video, he talks about the gap between prediction and accuracy. That is the “noise” in the communication model. And, just like in human communication, humans naturally fight to find meaning through that noise, so we tolerate the imperfections and the inaccuracies that AI gives us. Also, just like in human communication, we tend to believe what we tell ourselves is there in that noise. All persuasion is self-persuasion, after all.
So, the true risk here is that the reader, user, or audience is going to invest too much self-persuasion into the meaning in the gaps than is safe. Because there is so much information that no one human being can know, it is impossible to know where the gaps are for every conversation, especially over time. That makes a user of AI less and less diligent, and more and more reliant upon flawed information, for sure.
But it also reinforces the egoism of the user as well. There is no way to tell if the AI is wrong, and therefore if the user is wrong, and there’s no incentive for the user to believe that they could be wrong. This compounds over time.
That’s the true warning.