AightBits

There’s a growing misconception that generative transformer-based Large Language Models (i.e., OpenAI’s GPT-4) and the systems built around them (i.e., OpenAI’s ChatGPT 4o) are approaching “AGI” (Artificial General Intelligence) levels. It’s understandable that people who haven’t delved too deeply into the architecture, or inner workings, of LLMs might think so; the output of the latest systems is very impressive. However, it’s important to understand how these models and systems actually work and their limitations.

The basic concepts of LLMs are actually simpler than many realize: LLMs break down data during training and find patterns in the training data. They can then generate output using these learned patterns and probability. While models can exhibit reasoning-like behavior, this does not establish that they understand or comprehend in any human sense, nor do their underlying weights learn from user input after they are trained without further training.

They can, however, adapt to information provided during use (inference), either by the user or mechanisms that can use data from other resources on-demand. In other words, their trained knowledge is static with a knowledge cutoff date, though additional information can be provided during inference.

Individual LLM inference requests are also generally stateless, meaning they do not independently maintain persistent memory or perform functions in between waiting for input without additional systems to manage these capabilities. Finally, they are not inherently reliable calculators or symbolic reasoning engines, but instead generate answers to math and logic problems using learned patterns and computations that can produce convincing results without consistently applying valid mathematical or logical procedures.

It could be that LLMs may one day be one part of a complete AGI system (which may need to include the ability to learn persistently without retraining, take input from multiple sources, and other abilities which are generally unavailable to standalone text-based LLMs). However, there is substantial reason to question whether scaling current LLM approaches alone will be sufficient to reach AGI.

I thought it might help to explain how these models are actually trained and how they function:

Training

1. Prior to training a model, a vast amount of data is collected and curated from many sources such as books, news, online repositories like Wikipedia, chat conversations, and much more. This data serves as the foundation for training the model. The resulting broadly trained models are often known as foundation models because they can be adapted for a wide range of downstream tasks, including through further training or fine-tuning.
2. The models’ weights are static during inference – i.e., they do not change or retain new information from being used without additional training or fine-tuning. To overcome the static nature of pretrained models and their knowledge cutoffs, mechanisms like Retrieval-Augmented Generation, function calling, and vector databases can be used to provide a static model with more in-context data during inference for a particular prompt, but these technologies do not actually alter the models’ weights.
3. During training, this data is broken down into tokens (words, parts of words, or even individual characters/symbols). Patterns in the training data are learned and encoded in numerical values called “weights” in this context. The total number of learned parameters is often referred to as the parameter count (i.e., a model with 8 billion parameters may be said to be an 8B model).
4. The pretrained model may now be further trained (fine-tuned) for a specific task. This is done in a number of ways but the concept is similar to pretraining. This time, the model may be fed additional data such as examples of how it should respond (i.e., to chat with a user, to follow instructions, etc.) or domain-specific data on a particular subject.

Now that we have a pretrained model, let’s go on to inference, or the process of an application or end user actually sending requests, or prompts, to the model:

Use (Inference)

1. Once training is complete, we have a pretrained model. At this point, this model can perform basic tasks like text completion (i.e., if you feed the model “A cat is” it might respond with “a furry mammal”) but is generally not ready to follow instructions or chat at this point. Note that some models may be trained to perform more complex tasks like instruction following and chat during the initial training.
2. At this point, the model can be loaded (i.e., through code or using a GUI that utilizes a library for loading and interacting with a model) and queried via prompts. This process is known as inference.
3. Before a prompt is processed by the model, it is broken down into tokens which are then converted into numerical vector representations, or embeddings. As these embeddings are processed through the model, they produce intermediate values known as activations, which are distinct from the model’s learned weights.
4. The model uses its learned weights and the activations produced from the input tokens to generate a response, token by token. It does this by processing the input through layers of learned numerical transformations to calculate a probability distribution for the next token, then selecting or sampling a token from that distribution and repeating until the response is complete. Some models may also generate intermediate reasoning steps before producing their final response.
5. These tokens are converted back into language and presented to the user.

Pitfalls

1. Generalized language is bland without providing your own data and careful prompting

By its nature, AI generated content is often very bland and formulaic, even across different models. This is partly because models are trained to generate probable text based on patterns in their training data, and many models share similar data sources and training objectives. However, the output is also affected by fine-tuning, prompting, and how tokens are selected during generation.

2. Static models, stale data, and date cutoffs:

Models are static during inference, meaning after a model is trained the weights (or parameters) do not learn any new information or change based on use (inference). For a model to permanently incorporate new information into its weights, it must be further trained through fine-tuning or other weight-updating methods.

Alternatively, systems can retain or retrieve information using external memory and methods like Retrieval-Augmented Generation (RAG), function calling, and agents which we will discuss later. These methods can provide access to new information without changing the underlying model.

3. Hallucinations:

Perhaps one of the most important and most misunderstood limitations of generative AI are hallucinations. Hallucinations are when the AI seemingly makes up facts or generates non-sensical or contradictory text. This may sound like a bug but it’s actually a normal occurrence of how LLMs generate text – not necessarily recalling stored knowledge verbatim but generating output through learned patterns and probability, without an inherent mechanism to guarantee factual correctness.

Generally, the more reliable and relevant information an LLM has learned about a particular subject during training, the more likely it is to generate a correct response. Conversely, gaps in training data, misleading patterns, ambiguity, or insufficient context can increase the likelihood of hallucinations.

This is a problem regardless of use case, but even more so in the areas of medicine, psychology, and law. There are many examples where LLMs have confidently presented fabricated case law to lawyers and harmful medical or therapy advice to practitioners and patients alike, so it’s important that consequential output is thoroughly verified.

4. Garbage In, Garbage Out (GIGO):

Errors in training data or user prompts can result in errors in the LLM’s output. For chat-based inference, where a user converses with a model over time, these errors can be compounded as the model may begin to incorporate the user’s flawed premises as well as its own previous incorrect responses into subsequent output, increasing the risk of further errors. While models can sometimes recognize and correct these errors, they cannot be relied upon to do so consistently.

5. Don’t anthropomorphize AI:

Except for certain use cases, like roleplay for world building, game development, fiction writing, and entertainment, anthropomorphizing can not only affect the quality of output but can have negative emotional impact as well. It’s important to remember that LLMs generate text based on mathematical computation, and there is no established evidence that they are sentient or have feelings. Their apparent “thinking” should not be confused with human cognition or subjective experience.

6. LLMs Don’t Reliably Perform Math or Apply Logic:

As the name Large Language Model suggests, LLMs are primarily language-oriented models and are not inherently reliable calculators or symbolic logic engines. They can generate correct solutions to complex math and logic problems, but this does not establish that they are consistently applying valid mathematical or logical procedures.

Their performance can depend heavily on learned patterns, and they can fail problems requiring reliable application of formal rules. Studies have shown that even models performing well on mathematical benchmarks can fail when problems are altered in ways that should not affect their underlying logic. These limitations may be addressed in part by methods like function calling.

For math problems as an example, models may be trained to identify math problems and then call an external tool like Wolfram Alpha to perform exact calculations or symbolic operations. Models can also use code execution tools for calculations, but these approaches can be flawed because of the complex chain (or pipeline) that must be followed.

For example, the model must identify a math problem, pass it to code that breaks the problem down into a format the tool can understand, pass along the problem, retrieve the solution, and then present it to the user. Like the old game of Telephone, the information may be miscommunicated at any point in the chain which may result in an incorrect answer.

Conclusion

In short, LLMs are powerful language tools, made more powerful when used with other tools such as GUIs and applications that can handle state management (remembering context between prompts), function calling and agents (to work with external data), vector databases and other memory systems (for retaining or retrieving information between sessions).

However, while these tools excel at language-specific tasks like summarization, categorization, translation, sentiment analysis, proofreading and editing, and similar, they have real limitations concerning factual reliability, robust reasoning, mathematical consistency, and non-language tasks. Although increasingly capable systems can perform tasks beyond language processing, these capabilities should not be confused with general intelligence or an absence of the underlying models’ limitations.

Whether current LLM architectures can be scaled to AGI remains technically uncertain but unlikely, and there is substantial expert skepticism that scaling these approaches alone will be sufficient. LLMs may ultimately be one component of more advanced AI systems rather than a complete solution themselves.