We hear many stories about large language models, a form of artificial intelligence, responding to queries with answers that range from amusing to horrifying. Amusing includes suggesting glue to hold pepperoni on pizza. Horrifying includes proposing suicide and the ways to accomplish it. Really erroneous answers are built in, according to a team from the Georgia Institute of Technology and from OpenAI, a huge presence in the world of AI. The simple reason is that AI is rewarded (trained) for never saying “I don’t know.” Many queries really are unanswerable. You can think of some, ranging from nonsense (How old is blue?) to unavailable data (Which is the biggest star?). We have to be aware that LLMs are scored, or benchmarked, to provide answers that are “acceptable” with enormous numbers of scores on individual queries. The scores are positive for a correct answer but only zero for an incorrect answer. There’s no penalty, no negative score, for a bad answer. Even LLMs trained to say “I don’t know” are likely to “game the system” to be rewarded, says Hao Peng at the University of Illinois. Finally, saying “I don’t know” would be financially devastating for the AI company. Users of the platform will quickly go to an alternative platform to get a definite, and wrong, answer. AI will always be flawed. Caveat emptor.
This has been an outreach activity of the Las Cruces Academy, viewable at GreatSchools.org
Source: Science, 6 Nov. 2025, pp. 558-9.
Image: My own photo of a pizza box
