Deep Learning for Text Processing: How Neural Networks Understand Language

From Words to Numbers: Tokenization and Embeddings

Neural networks operate on numbers, not words. The first step in any text processing pipeline is converting text into numerical representations. Tokenization splits text into discrete units called tokens. Early systems used word-level tokenization, where each unique word in the vocabulary gets its own integer ID. The sentence "The cat sat" becomes something like [42, 891, 1203]. This works but creates problems: rare words have few training examples, misspellings become unknown tokens, and the vocabulary can grow to hundreds of thousands of entries.

Modern systems use subword tokenization, which breaks words into smaller meaningful pieces. The Byte Pair Encoding (BPE) algorithm, used by GPT models, starts with individual characters and iteratively merges the most frequent adjacent pairs. Common words like "the" remain as single tokens, while rare words like "neuroscience" are split into subwords like "neuro" and "science." This balances vocabulary size (typically 30,000 to 100,000 tokens) with the ability to represent any word, including words never seen during training, through subword combinations.

Token IDs are then converted into dense vectors called embeddings. Each token is looked up in an embedding table, a matrix where each row is a learned vector (typically 256 to 4096 dimensions) representing that token. These embeddings are learned during training, and tokens with similar meanings end up with similar vectors. The embedding for "king" is close to the embedding for "queen," and the vector difference between "king" and "queen" is similar to the difference between "man" and "woman." This geometric structure in embedding space captures semantic relationships that the model uses for all downstream processing.

The RNN Era: Sequential Processing

Before transformers, recurrent neural networks processed text one token at a time, maintaining a hidden state that accumulated context from all previous tokens. LSTMs and GRUs, the practical RNN variants, could maintain relevant context across roughly 100 to 200 tokens, enough for single sentences and short paragraphs. The encoder-decoder architecture with attention, introduced for machine translation in 2014, used an RNN encoder to process the source sentence and an RNN decoder with attention to generate the translation one word at a time.

RNN-based systems achieved the first major breakthroughs in neural machine translation, text generation, and speech recognition. Google replaced its phrase-based statistical translation system with an LSTM-based neural system in 2016, producing noticeably more fluent translations. By 2017, neural machine translation was standard for all major language pairs. These systems were good, but they were limited by the sequential nature of RNN processing: training was slow because each token had to wait for the previous token's computation to finish, and very long documents were beyond practical reach.

The Transformer Revolution

The transformer architecture, applied to text through models like BERT and GPT, eliminated the sequential bottleneck by processing all tokens simultaneously through self-attention. This had two immediate effects: training became dramatically faster (because all positions could be computed in parallel on GPUs), and the model could directly relate any two tokens regardless of their distance in the text, removing the long-range dependency problem that plagued RNNs.

BERT (2018) demonstrated that a transformer encoder trained to predict masked words in sentences learned deep language understanding that transferred to virtually any text task. Fine-tuning BERT on a few thousand labeled examples set new records on 11 NLP benchmarks simultaneously. The model learned syntax, semantics, coreference, and even some world knowledge purely from predicting missing words in a large text corpus. This pre-train-then-fine-tune paradigm became the standard approach for text processing.

GPT models showed that a transformer decoder trained to predict the next token could generate coherent, contextually appropriate text of arbitrary length. GPT-2 (2019) generated convincing news articles and stories. GPT-3 (2020) could perform tasks from translation to question answering without any fine-tuning, simply by being given a few examples in the prompt. GPT-4 and subsequent models extended these capabilities to reasoning, code generation, and multi-step problem solving, capabilities that emerged from scale rather than being explicitly designed.

Core Text Tasks

Text Classification

Text classification assigns labels to documents: spam vs legitimate, positive vs negative sentiment, topic categories, language identification, or intent detection for virtual assistants. A fine-tuned BERT model typically achieves 90 to 95% accuracy on sentiment classification benchmarks with just a few thousand training examples. For simpler tasks like language identification or spam detection, even smaller models achieve near-perfect accuracy. The practical pipeline is straightforward: take a pre-trained language model, add a classification head (a single linear layer), and fine-tune on labeled examples.

Machine Translation

Modern neural machine translation uses encoder-decoder transformers that process the entire source sentence and generate the translation token by token. The quality for major language pairs (English-French, English-German, English-Chinese) approaches professional human translation for straightforward text. BLEU scores, the standard automated metric, have roughly doubled compared to pre-neural systems. The remaining challenges are idiomatic expressions, cultural context, ambiguity, and low-resource languages where training data is scarce. Multilingual models like NLLB (No Language Left Behind), trained on 200 languages, have extended decent translation quality to many languages that previously had minimal MT support.

Text Generation

Autoregressive language models generate text by predicting one token at a time, each conditioned on all previously generated tokens. The quality depends on model size, training data, and sampling strategy. Greedy decoding (always picking the most probable next token) produces bland, repetitive text. Temperature sampling introduces controlled randomness: lower temperature produces more focused, predictable text, while higher temperature produces more creative, varied output. Top-p (nucleus) sampling restricts the token pool to the smallest set whose cumulative probability exceeds a threshold, balancing diversity and coherence.

Summarization

Extractive summarization selects the most important sentences from a document. Abstractive summarization generates a new, shorter text that captures the key points in the model's own words. Modern transformer models excel at abstractive summarization because they can process the entire document through self-attention and generate a condensed version that reorganizes and paraphrases the content. Long-context models that handle 100,000+ tokens can summarize entire books, though quality degrades for very long inputs where the model must compress a large amount of information into a short summary.

Question Answering

Extractive QA identifies the span of text within a document that answers a given question. The model reads both the question and the document, and outputs the start and end positions of the answer span. SQuAD (Stanford Question Answering Dataset) is the standard benchmark, and modern models achieve F1 scores above 93%, exceeding average human performance. Open-domain QA systems retrieve relevant documents from a large corpus and then extract or generate answers, combining information retrieval with language understanding. Retrieval-augmented generation (RAG) has become the standard architecture for knowledge-grounded question answering.

Practical Considerations

Text length is a critical constraint. Most transformer models have a fixed context window: BERT handles 512 tokens (roughly 400 words), GPT-3 handles 4,096 tokens, and newer models handle 128,000 to 1,000,000+ tokens. For documents longer than the context window, you must either truncate (losing information), chunk and process separately (losing cross-chunk context), or use a model with a sufficiently large context window. The quadratic memory cost of self-attention means that longer contexts require proportionally more GPU memory.

Fine-tuning versus prompting is a key practical decision. Fine-tuning adapts a pre-trained model to your specific task by training on labeled examples, producing a specialized model. Prompting provides instructions and examples directly in the input text, using the model's general capabilities without any additional training. Fine-tuning produces better performance for well-defined tasks with available training data. Prompting is more flexible, requires no training data, and works with commercial API models where you cannot modify the weights. For many applications, the best approach combines both: fine-tune a base model on your domain data, then use prompting to handle the variety of specific requests at inference time.

Evaluation of text processing systems requires careful attention to metrics. Accuracy works for classification but not for generation tasks. BLEU and ROUGE measure overlap between generated text and reference texts, but they correlate poorly with human judgments of quality for open-ended generation. Human evaluation remains the gold standard for generation tasks, but it is expensive and slow. LLM-as-judge approaches, where one language model evaluates the output of another, are increasingly used as a scalable proxy for human evaluation.