How Does AI Improve Over Time?

More Data, Better Data

The simplest way to improve an AI model is to give it more training data. Performance on most tasks follows a power law with dataset size: doubling the data does not double the accuracy, but it reliably improves it by a consistent, predictable amount. This relationship has held across image classification, language modeling, speech recognition, and dozens of other tasks.

Quality matters as much as quantity. A common improvement cycle starts by deploying a model, collecting examples where it fails, adding those failure cases to the training set, and retraining. This process, sometimes called data flywheel, means the model gets better at exactly the cases it previously struggled with. Tesla's self-driving system uses this approach: when a car encounters an unusual situation, the data is flagged and can be incorporated into future training.

Data cleaning and curation also drive improvement. Removing duplicates, filtering low-quality examples, and correcting label errors can improve a model's performance without adding any new data. Researchers at Google found that carefully curating their training data for PaLM 2 produced a model that outperformed the larger PaLM despite having fewer parameters, demonstrating that data quality can compensate for model size.

Scaling Up

In 2020, OpenAI published research on scaling laws showing that language model performance improves as a smooth power law of three factors: the number of parameters, the size of the training dataset, and the amount of compute used for training. These laws are remarkably predictable, allowing researchers to estimate how a model will perform before training it by extrapolating from smaller experiments.

This finding drove the rapid increase in model size from 2020 onward. GPT-2 (1.5 billion parameters) to GPT-3 (175 billion) to GPT-4 (estimated over a trillion) represented orders-of-magnitude increases in scale, and each jump brought substantial capability improvements. The scaling laws predicted these improvements accurately.

However, scaling has diminishing returns and increasing costs. Each halving of the error rate requires roughly an order of magnitude more compute. Training costs scale faster than linearly with model size because larger models need more data, more memory, and more sophisticated distributed training infrastructure. At some point, the cost-performance tradeoff favors architectural innovation over pure scaling.

Architectural Innovation

New architectures can provide step-function improvements that scaling alone cannot achieve. The transformer, introduced in 2017, was not just an incremental improvement over RNNs. It was a fundamentally different approach that unlocked capabilities impossible with the previous architecture, regardless of scale.

Mixture of Experts (MoE) is a more recent architectural innovation. Instead of using all parameters for every input, MoE routes each input to a subset of specialized "expert" sub-networks. This allows models with trillions of total parameters to be computationally efficient because only a fraction of parameters are active for any given input. GPT-4 is believed to use an MoE architecture, which explains how it can have a very large total parameter count while remaining fast enough for real-time interaction.

Other architectural improvements include flash attention (which reduces the memory cost of self-attention from quadratic to near-linear), rotary position embeddings (which improve how models handle sequence position), and grouped query attention (which reduces inference cost by sharing key and value projections across attention heads). Each of these innovations improves either performance, efficiency, or both.

Human Feedback Loops

Reinforcement Learning from Human Feedback (RLHF) was the breakthrough that made language models useful as conversational assistants. But human feedback continues to drive improvement after initial deployment.

Users who flag bad responses provide direct signal about failure modes. Conversations where users rephrase their request (implying the first answer was unsatisfactory) provide implicit signal. Red teams deliberately probe the model for safety failures, generating training data for future improvements. All of this feedback can be incorporated into the next training run.

Constitutional AI, developed by Anthropic, extends this approach by having the AI model evaluate its own outputs against a set of principles, reducing the need for human annotators. The model generates a response, critiques it against principles like "be helpful" and "be harmless," revises it, and the revision pair becomes training data. This creates a scalable feedback loop that can improve safety and helpfulness without requiring a human to evaluate every response.

Continual and Online Learning

Most deployed AI systems are static: they are trained once and their parameters are frozen during deployment. But research into continual learning aims to create systems that keep learning from new data without forgetting what they already know.

The main obstacle is catastrophic forgetting. When a neural network is trained on new data, it tends to overwrite the parameters that stored knowledge from previous training. A model that learns a new language might forget its old language in the process. Techniques like elastic weight consolidation (which penalizes changes to parameters that were important for previous tasks) and progressive neural networks (which add new capacity for new tasks while freezing old parameters) partially address this, but no solution is fully satisfactory yet.

Online learning, where the model updates with each new example, is used in some applications like recommendation systems and spam filters where the data distribution changes rapidly. These systems need to adapt to new spam patterns, new user preferences, and new products in real time. The tradeoff is that online learning is less stable than batch training and requires careful engineering to avoid drift.

Evaluation and Benchmarks

Improvement requires measurement, and the AI field uses benchmark suites to track progress. MMLU (Massive Multitask Language Understanding) tests knowledge across 57 academic subjects. HumanEval tests code generation ability. HellaSwag tests common-sense reasoning. Each benchmark captures a different dimension of capability.

Benchmarks have limitations. Models can improve on a benchmark without improving in ways that matter to real users, a phenomenon called benchmark hacking. Conversely, real improvements in helpfulness or safety may not show up on any standard benchmark. The field is increasingly moving toward human evaluation (asking people to judge model outputs) alongside automated benchmarks, recognizing that both are necessary for a complete picture.