Spiking Neural Networks Explained

How Biological Neurons Inspire SNNs

Real neurons in the brain communicate through action potentials, brief electrical pulses that travel along axons to reach other neurons. A biological neuron integrates incoming signals at its synapses, and when the accumulated voltage crosses a threshold (about -55 millivolts), it fires a spike that propagates to all connected neurons. After firing, the neuron enters a refractory period where it cannot fire again for a few milliseconds.

Conventional artificial neural networks abstract away this spiking behavior entirely. They pass continuous values (like 0.73) between neurons at each computation step. This simplification makes them easier to train but discards the temporal dynamics that biological neurons use. SNNs retain the spiking mechanism, encoding information in the timing and frequency of spikes rather than in continuous activation values.

The Leaky Integrate-and-Fire Model

The most common SNN neuron model is the leaky integrate-and-fire (LIF) neuron. It maintains a membrane potential that changes over time according to three rules:

Integration. Incoming spikes increase the membrane potential by the synaptic weight of the connection. If a neuron receives a spike from a connection with weight 0.3, its membrane potential increases by 0.3.

Leakage. The membrane potential decays toward a resting value over time, simulating the biological leakage of charge through the cell membrane. Without incoming spikes, the potential gradually returns to zero. The leak rate is a parameter that controls how quickly the neuron "forgets" past inputs.

Firing. When the membrane potential crosses a threshold, the neuron emits a spike (a binary 1 transmitted to all downstream neurons). After firing, the potential resets to the resting value, and the cycle begins again.

This simple model captures the essential dynamics of biological neurons while remaining computationally tractable. More detailed models (Hodgkin-Huxley, Izhikevich) add biological realism at the cost of computational complexity, but the LIF model is sufficient for most SNN applications.

Temporal Coding

SNNs can encode information in multiple ways, all involving the timing of spikes. Rate coding uses the frequency of spikes: more spikes per unit time represents a stronger signal, similar to how conventional networks use larger activation values. Temporal coding uses the precise timing of individual spikes: earlier spikes represent stronger or more important features. Population coding uses the pattern of spikes across a group of neurons to represent complex information.

Temporal coding is particularly interesting because it allows SNNs to process information faster than rate coding. A single precisely timed spike can convey as much information as a firing rate estimated over many spikes. This is consistent with neuroscience evidence that biological neural circuits often make decisions based on the first spikes to arrive rather than waiting for a firing rate to stabilize.

Training Challenges

The fundamental challenge of SNNs is training. Backpropagation, the workhorse of conventional neural network training, requires continuous, differentiable functions. Spikes are binary events (0 or 1) with a step-function relationship to the membrane potential, which has a derivative of zero everywhere except at the threshold, where it is undefined. This non-differentiability prevents direct application of standard backpropagation.

Several approaches work around this limitation:

Surrogate gradients. Replace the non-differentiable spike function with a smooth approximation during the backward pass while keeping the actual spike in the forward pass. The surrogate (typically a sigmoid or piecewise linear function) provides non-zero gradients that guide weight updates. This is the most successful approach currently, achieving results on image classification benchmarks within a few percentage points of conventional networks.

ANN-to-SNN conversion. Train a conventional neural network with standard backpropagation, then convert the trained weights to an SNN by replacing continuous activations with spike rates. This avoids SNN training entirely but produces SNNs that require many time steps to represent activation values as spike rates, reducing the energy efficiency advantage.

Spike-timing-dependent plasticity (STDP). A biologically inspired learning rule where synaptic weights change based on the relative timing of pre-synaptic and post-synaptic spikes. If a pre-synaptic spike consistently arrives just before the post-synaptic neuron fires, the connection is strengthened. STDP is unsupervised and local (each synapse only needs information from its two connected neurons), making it biologically plausible but less powerful than backpropagation-based training.

Neuromorphic Hardware

SNNs achieve their energy efficiency advantage only on hardware designed for event-driven computation. Conventional GPUs execute floating-point operations continuously whether the values are zero or not. Neuromorphic chips process spikes only when they occur, consuming energy proportional to the number of spikes rather than the number of neurons.

Intel's Loihi 2 implements 128 neuromorphic cores, each supporting up to 8,192 neurons with programmable synaptic learning rules. It achieves 100x better energy efficiency than GPUs for certain SNN workloads. Loihi 2 has been used for robotic control, olfactory sensing, and optimization problems.

IBM's TrueNorth implements 1 million neurons and 256 million synapses on a single chip consuming only 70 milliwatts. It demonstrated real-time object detection from video at a fraction of the power consumption of GPU-based approaches.

SynSense's Xylo and similar chips target edge computing applications where power consumption is critical: hearing aids, wearable sensors, always-on keyword detection, and IoT devices. These chips process audio and sensor data using SNNs at microwatt power levels.

Current State and Future Potential

SNNs currently trail conventional neural networks by 2 to 5 percentage points on standard benchmarks (ImageNet classification, CIFAR-10, speech recognition). This gap has narrowed significantly in recent years as surrogate gradient methods have improved, but it remains wide enough that SNNs are not competitive for applications where accuracy is the primary concern.

The case for SNNs rests on energy efficiency and biological plausibility rather than accuracy. For applications where power consumption is the binding constraint (always-on sensors, battery-powered devices, space-constrained hardware), SNNs on neuromorphic hardware offer 10-1,000x better energy efficiency than conventional networks on GPUs. As edge AI becomes more prevalent, this advantage becomes increasingly valuable.

SNNs may also prove important for understanding the brain. By studying what kinds of computations SNNs can and cannot perform, researchers gain insights into the computational principles of biological neural circuits. The interplay between SNN research and computational neuroscience benefits both fields.