Librarian's guide to Artificial Intelligence

What is a Language Model?

A massive language model, like the GPT algorithm, is a machine learning model trained to understand and produce language. It can read, listen, and respond fluently in nearly all natural languages and also understands several programming languages.

What does it "consume"?

Generative algorithms are trained using vast amounts of digital data. For a language model, this means it processes and learns from an enormous volume of written text. In the case of the most extensively used model, GPT-4, this includes practically the entire content of the internet.

How does a language model understand and respond?

The GPT language model, which powers ChatGPT and Copilot, uses probabilities to predict the content and structure of subsequent words and sentences. The model has absorbed millions of texts, learning the structures of languages, grammatical rules, and the information contained within the texts. The AI isn’t programmed to function in a specific way and doesn’t rely on a separate database; rather, its "intelligence" is based on the neural network it builds through learning. Thus, comparing the model’s function to human brains isn’t entirely far-fetched.

While AI can generate very human-like text, its "understanding" is based on statistical models, not genuine understanding or consciousness.

The language model does not understand language the way humans do:

- No genuine contextual understanding: AI does not deeply understand topics.

- No emotional understanding: AI can simulate emotions but does not experience them.

- Statistical, not logical understanding: The model's understanding is based on the frequency of words and their forms.

- No self-awareness or intention: AI does not possess self-awareness or its own volition.

- Limited context: Language models lack long-term memory and do not grasp broader conversational histories.

Challenges in Training Massive AI Models

- Finding relevant training material: One of the biggest challenges in creating newer and more expansive or specific models is locating sufficient, relevant, and usable data. If the data a model learns from is biased, the model's responses are likely to be biased as well.

- Computational power: Training generative models requires substantial computational power, necessitating more powerful supercomputers (GPU or TPU hardware).

- Energy consumption: Large models consume significant amounts of electrical energy, both during operation and training, leading to considerable environmental impacts.

- Model fine-tuning and potential distortions: To function as intended, generative models require precise fine-tuning and testing. As these models are akin to "black boxes," adjusting parameters and training the model may need alignments that can unpredictably distort the model’s conclusions. Fine-tuning for user-friendliness or achieving more morally acceptable responses could introduce unexpected biases in the model’s outputs.

ChatGPT is designed for interactive dialogue. It responds to user questions in natural language. ChatGPT has limited short-term memory, enabling it to produce coherent responses considering the conversation history. The model can "learn" during a single chat session, but it only retains this knowledge within that session.

Even though the GPT model can produce text that appears human-like and even profound, its "understanding" of language and text is fundamentally based on statistical models and not genuine comprehension or consciousness. Here are a few reasons why one might argue that GPT doesn't understand language in the same way humans do:

1. No Real Contextual Understanding: While GPT can form sentences on topics presented to it, it doesn't deeply understand these subjects. It can answer questions based on its statistical models and training data, but it doesn't grasp context in the same way humans do, who base their understanding on experiences, emotions, and cumulative life knowledge.

2. No Emotional Understanding: GPT can simulate emotions in its text, but it doesn't truly feel emotions. Human linguistic understanding often ties with their emotions, but GPT merely reacts to input without any emotional response.

3. Statistical, Not Logical Understanding: GPT's "understanding" is based on the frequency of words and phrases in its training data. It doesn't comprehend through logical contemplation or deductive thinking but rather reflects the frequency with which certain words or phrases appeared together in its training data.

4. No Self-awareness or Intent: GPT isn't self-aware and doesn't possess its own will. When a human communicates, they often have a purpose or intent behind their words. GPT responds based on input without any underlying intent or purpose.

5. Limited Context: While GPT can consider the context provided at a specific moment (like previous sentences in a conversation), it lacks long-term memory and doesn't grasp broader conversation histories or more holistic contexts.