AightBits

I posted this to a large Facebook AI group over a year ago after seeing 3-4 examples of Dunning-Kruger relating to LLMs in the same week. I had been on the fence about posting it, realizing it might come across as condescending or alarmist to some, and that wasn’t my intent. While not perfect, I still think there’s some value here so I’ll go ahead and post it here as well. I hope it helps.

The Dunning-Kruger Effect refers to a cognitive bias in which people with limited knowledge on a subject overestimate their competence with undue confidence, often rejecting the input of those who actually have knowledge and experience.

Unfortunately, there have been some concerning posts here [in this Facebook AI group] lately concerning the function and capabilities of Large Language Models like ChatGPT. More than just demonstrating a misunderstanding of how LLMs function, these posts give clear examples of how LLMs can foster the Dunning-Kruger Effect by creating echo chambers that reinforce user bias.

First, it’s important to understand that LLMs generate text based on language patterns learned during training from a vast amount of textual data, using probability and a small windows of user input (context). LLMs don’t comprehend the data they’ve learned or the user’s input — they use math and probability to create a response, one piece of a time and in sequential order. This makes LLMs very powerful language tools, but their advanced nature and convincing output often fool people into believing they actually have true cognitive abilities, and they absolutely do not.

There are three main factors that reinforce the Dunning-Kruger Effect, particular in those who straddle the line between casual LLM user and people with moderate technical understanding:

User-Induced Bias: LLMs function by taking a relatively small amount of user-provided context and running statistical calculations on that context and the model’s learned language patterns to generate new text using probability. The user’s context carries much more importance — or, more heavily influences probability — than the model’s own generalized knowledge. The more context a user provides (either through large prompts or long chat sessions), the more user bias increases. This may result in an echo chamber that reinforces incorrect or even dangerous beliefs.

Hallucinations: Hallucination refers to the possibility of LLMs making up facts, which can often sound very plausible and convincing, due to the fact they generate text by using learned patterns and language and probability, not by storing and recalling verified or curated knowledge verbatim.

Erroneous Information in Training Data: LLMs are trained on a vast amount of textual data, which can cause incorrect information to be generated for several reasons: 1) There may be mistakes in the training data, as much of it comes from unverified sources like social media, and 2) LLMs are trained on data spanning many generations, so some data may be out of date, and 3) LLMs are trained on fiction and literature, not just knowledge. Because LLMs generate text based on language patterns and not an understanding of where the source data may come from, it may mix both accurate and inaccurate/fictional information in output.

Now enter a user who has a significant gap between perceived confidence and actual expertise with LLMs. Most of what they know comes from their own experience with a convincing and seemingly confident model. They discuss their views and beliefs with the model, which may at first try to guide the user towards more factual information.

However, as the user presses or expands on their views, flaws in their input accumulate. As the LLM begins to generalize, probability shifts and the LLM slowly begins to echo back the user’s beliefs, attitudes, and even tone of voice. Because of human nature, this positive reinforcement and validation releases dopamine, and the cycle restarts. Rather than educating the user, the user may sometimes unintentionally influence the LLM.

Let’s take this one step further: A more advanced user may know how to create a custom “GPT” in ChatGPT. This added layer, which sits on top of the core model, may now be compromised from their insufficient understanding of how LLMs function. They believe they have created a more advanced, more aware version of the model, when in fact they’ve actually coerced it into agreeing with their misconceptions. The echo chamber is complete. What’s worse, this user has the ability to share their customization with others, extending the echo chamber.

Here are some simple strategies if you want to get the most objective and unbiased output from an LLM:

• Start with a fresh session for each new topic, or even subtle shift in topic, to prevent unwanted influence from older, unrelated, and/or potentially biased context .
• Ask the LLM for an objective, critical, and unbiased view or review.
• Present your own opinions in the third person. Some LLMs show more bias in favor of the user compared to an unknown source.
• Do not use any language that appears to support or refute the question or material.
• Do not present any unverified beliefs or assumptions as fact. Be honest about what is fact and what is conjecture.
• Factcheck ALL LLM output. Take nothing for granted.

How LLMs work, how to use them safely and effectively, and their potential pitfalls are beyond the scope of this post. I’ve posted on these topics many times if you want to read back, but there are many other sources as well.

Understanding the nature of generative AI is critical, and it’s important is that you don’t simply trust your own interactions with AI or the output from an LLM itself. Learning how LLMs are trained, how they function, and how to prompt in an objective, neutral way will help avoid this very sneaky trap.