How AI Understands Language

The Fundamental Problem: Words Are Not Numbers

Computers operate on numbers. Every computation inside a processor is arithmetic: addition, multiplication, comparison. But language is symbolic. The word "cat" has no inherent mathematical relationship to the word "dog" unless you define one. The first challenge in making AI understand language is converting symbols into numbers in a way that preserves meaning.

Early approaches used one-hot encoding, representing each word as a vector with a single 1 and thousands of zeros. The word "cat" might be [0, 0, 1, 0, 0, ...] and "dog" might be [0, 0, 0, 1, 0, ...]. This works for identification, but it tells the model nothing about how the words relate. The distance between "cat" and "dog" is the same as between "cat" and "democracy" because every pair of one-hot vectors is equally far apart. The encoding throws away all semantic information.

The solution is to learn the numbers rather than assign them arbitrarily. This is the idea behind word embeddings.

Word Embeddings: Learning What Words Mean

Word embeddings represent each word as a dense vector of perhaps 300 to 1024 dimensions, where the values are learned from data. The training process adjusts these vectors so that words appearing in similar contexts end up with similar vectors. Because "cat" and "dog" both appear near words like "pet," "fur," "veterinarian," and "food," their vectors converge to nearby points in the embedding space.

Word2Vec, introduced by Tomas Mikolov at Google in 2013, demonstrated this idea at scale. The model was trained on a simple task: given a word, predict the words around it (or vice versa). The prediction task itself was not particularly useful, but the learned word vectors turned out to encode rich semantic relationships.

The famous example is the arithmetic of embeddings. The vector for "king" minus "man" plus "woman" produces a vector closest to "queen." This works because the embedding space encodes the gender relationship as a consistent direction. Similarly, "Paris" minus "France" plus "Italy" gives a vector near "Rome." The model learned geography, gender relationships, and analogical reasoning purely from word co-occurrence patterns, without anyone telling it what countries, genders, or capitals are.

GloVe (Global Vectors for Word Representation), developed at Stanford in 2014, achieved similar results using a different training method based on word co-occurrence matrices rather than prediction tasks. Both approaches converge on the same insight: distributional semantics works. Words that appear in similar contexts have similar meanings, and this principle, when applied at scale, produces remarkably useful representations.

The Context Problem

Static word embeddings have a fundamental limitation: each word gets one vector regardless of context. The word "bank" has the same representation whether it appears in "river bank" or "investment bank." Human language is full of ambiguity that context resolves, and a single vector per word cannot capture this.

This limitation drove the development of contextual embeddings, where the representation of a word changes depending on the sentence it appears in. ELMo (Embeddings from Language Models), introduced in 2018 by researchers at the Allen Institute for AI, was the first major contextual embedding model. It used a bidirectional LSTM (a type of recurrent neural network) to process text in both directions, producing a different vector for "bank" in "river bank" than in "investment bank."

ELMo improved performance on nearly every natural language processing benchmark the day it was released, demonstrating that contextual representations are categorically better than static ones for almost every task. But the real breakthrough came a few months later with the transformer architecture.

Transformers and Self-Attention

The transformer, introduced in 2017, replaced recurrent processing with self-attention. Instead of reading text one word at a time and maintaining a running state, the transformer lets every word attend to every other word simultaneously. This parallel processing is both faster and more effective at capturing long-range dependencies.

Self-attention works by computing three vectors for each word: a query, a key, and a value. The query of one word is compared against the keys of all other words to produce attention weights. These weights determine how much each word contributes to the representation of the current word. The weighted sum of value vectors becomes the new contextual representation.

For example, in the sentence "The animal didn't cross the street because it was too tired," the word "it" needs to be linked to "animal" to understand the sentence correctly. The attention mechanism learns to assign a high weight between "it" and "animal" and a low weight between "it" and "street," effectively resolving the pronoun reference through learned statistical patterns.

Modern transformers use multi-head attention, running several attention computations in parallel with different learned parameters. Different heads can specialize in different types of relationships: one head might track syntactic dependencies (subject-verb agreement), another might track coreference (what pronouns refer to), and another might capture semantic similarity. The model learns these specializations automatically during training.

BERT and Bidirectional Understanding

BERT (Bidirectional Encoder Representations from Transformers), released by Google in 2018, applied the transformer to language understanding tasks. Its key innovation was a training objective called masked language modeling: randomly hiding 15% of the words in a sentence and training the model to predict them from context.

Unlike GPT, which reads text left-to-right, BERT reads in both directions simultaneously. To predict a masked word, BERT can use both the words before and after the gap. This bidirectional context gives BERT stronger representations for tasks like question answering, sentiment analysis, and text classification, where understanding the full sentence matters more than generating the next word.

BERT's pre-trained representations can be fine-tuned for specific tasks with very little task-specific data. A BERT model pre-trained on general text can be adapted to detect spam, classify legal documents, or extract medical entities with just a few hundred labeled examples. This transfer learning capability transformed the field because it meant you no longer needed massive task-specific datasets for every new application.

What AI Actually Learns About Language

Researchers have probed the internal representations of language models to understand what they learn. The findings reveal a surprisingly structured internal world.

Lower layers in the network tend to encode surface-level features: part of speech, word morphology, and local syntax. Middle layers encode syntactic structure: subject-verb relationships, clause boundaries, and dependency parses. Upper layers encode semantic meaning: topic, sentiment, entailment relationships, and factual associations. This hierarchy mirrors (loosely) how linguists decompose language into phonology, syntax, and semantics, though the model was never taught these categories.

Attention patterns in trained models reveal learned linguistic knowledge. Specific attention heads consistently track syntactic relationships. One head might always connect verbs to their subjects, even across complex sentences with multiple clauses and embedded phrases. Another head might track coreference chains, linking pronouns to their antecedents across long passages.

Models also learn factual knowledge. When prompted with "The capital of France is," a well-trained language model assigns high probability to "Paris" because it has seen this association thousands of times in training data. This factual knowledge is encoded implicitly in the model's parameters, distributed across millions of weights rather than stored in a lookup table.

The Limits of Statistical Understanding

Despite these capabilities, AI language understanding is fundamentally different from human understanding. Models operate on statistical associations, not grounded meaning. A model that knows "Paris is the capital of France" does not understand what a country is, what a capital city does, or what France looks like on a map. It knows that those words appear together in particular patterns.

This distinction matters for reliability. Statistical patterns are right most of the time but fail in specific, predictable ways. Models struggle with negation ("The king of France is not bald" still activates representations associated with baldness), novel combinations ("a dog that is also a prime number"), and reasoning that requires tracking multiple variables or applying rules consistently.

Models also reflect the biases in their training data. If the training text associates certain professions with certain genders more frequently, the model's embeddings will encode those associations as if they were facts about language. Debiasing techniques exist, but they address symptoms rather than the fundamental issue: statistical learning from text captures correlation, not truth.