Deep Reinforcement Learning: How AI Learns by Trial and Error

The Reinforcement Learning Framework

Reinforcement learning models decision-making as a loop between an agent and an environment. At each time step, the agent observes the current state of the environment, selects an action, and receives a reward signal along with the next state. The agent's goal is to learn a policy, a mapping from states to actions, that maximizes the cumulative reward over time. The policy must balance immediate rewards against long-term consequences: a move that wins a small advantage now might lead to a losing position later.

The key challenge is the credit assignment problem: when the agent receives a reward (or penalty), it must figure out which of its past actions contributed to that outcome. In a chess game that lasts 40 moves, a reward is given only at the end (win, lose, or draw). The agent must learn which specific moves in which positions were good and which were bad, a needle-in-a-haystack problem when the game tree has more possible positions than atoms in the universe.

The discount factor gamma controls how much the agent values future rewards relative to immediate ones. With gamma = 0, the agent is entirely myopic, caring only about the next reward. With gamma = 0.99, the agent values a reward 100 steps in the future at 37% of its immediate value, encouraging long-term planning. The choice of gamma is a critical hyperparameter that shapes the behavior the agent learns: too low and it becomes short-sighted, too high and training becomes unstable because the learning signal is diluted across too many time steps.

Deep Q-Networks (DQN)

The deep Q-network, introduced by DeepMind in 2013, was the first major success of deep reinforcement learning. DQN learned to play 49 Atari video games from raw pixel input, achieving human-level or superhuman performance on many games without any game-specific knowledge. The agent saw only the screen pixels and the score, and figured out how to play entirely through trial and error.

A Q-network is a neural network that takes a state as input and outputs a Q-value for each possible action. The Q-value estimates the total future reward the agent will receive if it takes that action and then follows the optimal policy afterward. At each step, the agent chooses the action with the highest Q-value (or a random action with small probability, for exploration). The network is trained by minimizing the difference between its predicted Q-values and the actual rewards received plus the discounted Q-value of the next state.

Two innovations made DQN work. Experience replay stores past transitions (state, action, reward, next state) in a buffer and trains on random batches from this buffer, breaking the temporal correlation between consecutive training samples that destabilizes learning. The target network is a slowly-updated copy of the Q-network used to compute the target values, preventing the moving-target problem where the network chases its own changing predictions. These techniques addressed fundamental instabilities in combining neural networks with Q-learning.

Policy Gradient Methods

While DQN learns which actions are best by estimating their values, policy gradient methods learn the policy directly. The neural network outputs a probability distribution over actions, and the agent samples from this distribution. The policy is updated by increasing the probability of actions that led to high rewards and decreasing the probability of actions that led to low rewards. This is done by computing the gradient of the expected reward with respect to the policy parameters and updating in that direction.

The REINFORCE algorithm is the simplest policy gradient method: run the agent through a complete episode, compute the total reward, and increase the probability of every action taken in proportion to the total reward. This works but has extremely high variance because the reward signal is noisy and the same action can lead to very different outcomes depending on the rest of the episode. Variance reduction techniques, including subtracting a baseline from the rewards and using advantage estimates instead of raw rewards, are essential for practical policy gradient methods.

Policy gradient methods have a key advantage over value-based methods: they can handle continuous action spaces. A robotic arm that needs to choose a specific angle for each joint has a continuous, high-dimensional action space where Q-learning would require discretizing the space (losing precision) or special architectures. Policy gradients output a continuous distribution (typically Gaussian) over action values, naturally handling this setting. This makes policy gradients the default approach for robotics and other continuous control tasks.

Actor-Critic Methods

Actor-critic methods combine the strengths of both approaches. The actor is a policy network that selects actions. The critic is a value network that evaluates how good the current state is. The critic's value estimates provide a low-variance baseline for the actor's policy gradient updates, while the actor provides the exploratory behavior that generates the training data for the critic. This combination reduces the variance of policy gradients while maintaining their ability to handle continuous actions.

Proximal Policy Optimization (PPO), developed by OpenAI in 2017, is the most widely used actor-critic algorithm. PPO constrains each policy update to be small, preventing the catastrophic performance collapses that occur when the policy changes too dramatically in a single step. The "clipped" objective function ensures that the probability ratio between the new and old policies stays within a narrow range (typically 0.8 to 1.2), providing a simple, stable, and effective training procedure. PPO is the algorithm behind many of the most visible RL successes, including OpenAI Five (Dota 2), ChatGPT's RLHF fine-tuning, and numerous robotics applications.

Soft Actor-Critic (SAC) adds an entropy bonus to the reward, encouraging the agent to maintain a diverse set of behaviors rather than committing to a single strategy. This helps with exploration (the agent tries more things) and robustness (the agent learns multiple ways to accomplish its goal). SAC is particularly popular for robotics because the entropy-encouraged diverse behavior produces more robust physical skills that transfer better from simulation to real hardware.

Landmark Achievements

AlphaGo (2016) defeated the world champion at Go, a game with more possible board positions than there are atoms in the observable universe. The system combined deep neural networks (for evaluating board positions and suggesting moves) with Monte Carlo tree search (for planning ahead). AlphaGo Zero (2017) went further, learning to play Go from scratch with zero human knowledge, starting from random play and achieving superhuman performance within 40 days through self-play alone. AlphaZero extended this to chess and shogi, achieving superhuman performance in all three games with the same algorithm.

OpenAI Five (2019) defeated the world champion team at Dota 2, a complex real-time strategy game with imperfect information, long time horizons (games last 45 minutes), and enormous action spaces. The system trained by playing the equivalent of 45,000 years of Dota 2 against itself, using massive-scale distributed RL with 256 GPUs and 128,000 CPU cores. This demonstrated that deep RL could handle far more complex and ambiguous environments than board games.

In the physical world, deep RL has controlled plasma in a tokamak fusion reactor (maintaining stable plasma shape through magnetic field adjustments), optimized Google's datacenter cooling (reducing energy consumption by 40%), and trained robotic hands to manipulate a Rubik's cube with remarkable dexterity. These applications demonstrate that RL can learn control policies for systems too complex to model analytically.

Challenges and Limitations

Sample efficiency is deep RL's biggest weakness. DQN required approximately 200 million frames (about 39 days of game time at normal speed) to learn each Atari game. OpenAI Five trained on 45,000 years of game time. Real-world applications cannot afford this level of trial and error, which is why most RL systems train in simulation and then transfer to the real world (sim-to-real transfer). Domain randomization, where the simulation's physics, visuals, and dynamics are varied during training, produces policies that are robust enough to work on real hardware despite the inevitable differences between simulation and reality.

Reward design is critical and often difficult. The agent optimizes exactly what you tell it to optimize, which is not always what you want. A robot rewarded for moving forward might learn to fall forward. A game agent rewarded for score might discover exploits that maximize score without actually playing well. Reward shaping, where intermediate rewards guide the agent toward desired behavior, and reward learning from human feedback (RLHF), where a model learns the reward function from human preferences, are approaches to this problem. RLHF has been particularly successful for aligning language models with human preferences.

Stability and reproducibility remain challenging. Small changes in hyperparameters, random seeds, or environment details can produce dramatically different outcomes. A training run that produces superhuman performance with one random seed might fail completely with another. This sensitivity makes RL research difficult to reproduce and RL systems difficult to deploy reliably. Careful hyperparameter tuning, multiple random seeds, and robust evaluation protocols are essential.