Most importantly, it’s useful to understand the purpose and context of weights and biases: the adjustable model parameters that are optimized through backpropagation and gradient descent.

Neural networks aim to roughly mimic the structure of the human brain. They’re composed of many interconnected nodes (or neurons), arranged in layers. Neural networks make predictions once the original input data has made a forward pass through the entire network.

Neurons in the input layer receive input data, usually as a vector embedding, with each input neuron receiving an individual feature of the input vector. For example, a model that works with 10x10 pixel grayscale images will typically have 100 neurons in its input layer, with each input neuron corresponding to an individual pixel. Neural networks thus typically require inputs of fixed size, though techniques like pooling or normalization can provide some flexibility.

In a standard feedforward neural network, each neuron in the input layer is connected to each of the neurons in the following layer, which are themselves connected to the neurons in the next layer, and so on until the output layer where final predictions are made. The intermediate layers between the input layer and output layer called the network’s hidden layers, are where most “learning” occurs.

While some specialized neural network architectures, such as mixture of expert models or convolutional neural networks, entail variations, additions or exceptions to this straightforward arrangement, all neural networks employ this core structure.

Though each neuron receives input from each node of the previous layer, not all of those inputs are given the same importance. Each connection between two neurons is given a unique weight: a multiplier that increases or decreases one neuron’s contribution to a neuron in the following layer.

Each individual neuron may also be given a bias: a constant value added to the sum of the weighted inputs from the neurons in the previous layer.

The ultimate goal of backpropagation and gradient descent is to calculate the weights and biases that will yield the best model predictions. Neurons corresponding to data features that significantly correlate with accurate predictions are given greater weights; other connections may be given weights approaching zero.

Modern deep neural networks, often with dozens of hidden layers each containing many neurons, might comprise thousands, millions or—in the case of most large language models (LLMs)—billions of such adjustable parameters.

Each neuron is configured to perform a mathematical operation, called an activation function, on the sum of varyingly weighted inputs it receives from nodes in the previous layer. Activation functions introduce nonlinearity, enabling the model to capture complex patterns in input data and yield gradients that can be optimized. Using only linear activation functions essentially collapses the neural network into a linear regression model.

Common activation functions in neural networks include:

• The sigmoid function, which maps any input to a value between 0 and 1.
• The hyperbolic tangent (or tanh) function, which maps inputs to a value between -1 and 1.
• The rectified linear unit (or ReLU), which maps any negative input to 0 and leaves any positive input unchanged.
• The softmax function, which converts a vector of inputs to a vector whose elements range from 0 and 1 and collectively sum to 1.

Consider a hypothetical hidden unit z, with a tanh activation function and bias term t, in the second layer of a neural network with 3 input nodes, a, b and c, in its input layer. Each of the connections between the input nodes and node z has a unique weight, w. We can describe the output value that node z will pass to the neurons in the next layer with the simplified equation z = tanh(w_az*a + w_bz*b + w_cz*c + t).

The neuron z is connected to neurons in the next layer. That equation for z is therefore part of the activation functions in the next layer and, by extension, also part of every activation function for any neurons in any subsequent layer.

Abstractly speaking, the purpose of backpropagation is to train a neural network to make better predictions through supervised learning. More fundamentally, the goal of backpropagation is to determine how model weights and biases should be adjusted to minimize error as measured by a loss function.

On a technical, mathematical level, the goal of backpropagation is to calculate the gradient of the loss function with respect to each of the individual parameters of the neural network. In simpler terms, backpropagation uses the chain rule to calculate the rate at which loss changes in response to any change to a specific weight (or bias) in the network.

Generally speaking, training neural networks with backpropagation entails the following steps:

• A forward pass, making predictions on training data.
• A loss function measures the error of the model’s predictions during that forward pass.
• Backpropagation of error, or a backward pass, to calculate the partial derivatives of the loss function.
• Gradient descent, to update model weights.

Neural networks output predictions through forward propagation. Forward propagation is essentially a long series of nested equations, with the outputs of the activation functions from one layer of neurons serving as inputs to the activation functions of neurons in the next layer.

Model training typically begins with a random initialization of weights and biases. Model hyperparameters, such as the number of hidden layers, the number of nodes in each layer and activation functions for specific neurons, are configured manually and not subject to training.

In each forward pass, an input is sampled from the training data set. The nodes of the input layer receive the input vector, and each passes their value—multiplied by some random initial weight—to the nodes of the first hidden layer. The hidden units take the weighted sum of these output values as input to an activation function, whose output value (conditioned by a random initial weight) serves as input to the neurons in the next layer. This continues until the output layer, where a final prediction occurs.

Consider this simplified example of a neural network that classifies inputs into one of 5 categories:

• The input layer receives a numerical representation of an example sampled from the training data.
• The input nodes pass their values to hidden units in the next layer. The hidden units use a ReLU activation function.
• Data flows through the hidden layers, each progressively extracting key features until it reaches the output layer.
• The output layer contains 5 neurons, each corresponding to a potential classification category.
• The output neurons use a softmax activation function. The output value of each output neuron’s softmax function corresponds to the probability, out of 1, that the input should be classified as the category that the neuron represents.
• The network predicts that the original input belongs to the category of whichever output neuron has the highest softmax value.

In a well-trained network, this model will consistently output a high probability value for the correct classification and output low probability values for the other, incorrect classifications. However, this neural network isn’t yet trained. At this point, its weights and biases have random initial values, so its predictions are generally inaccurate.

After each forward pass, a loss function measures the difference (or “loss”) between the model’s predicted output for a given input and the correct predictions (or “ground truth”) for that input. In other words, it measures how different the model’s actual output is from the desired output.

In supervised learning, which uses labeled data, ground truth is provided by manual annotations. In self-supervised learning, which masks or transforms parts of unlabeled data samples and task models by reconstructing it, the original sample serves as ground truth.

The goal of this loss function is to quantify inaccuracy in a way that appropriately reflects both the nature and magnitude of the error of the model’s output for each input. Different mathematical formulas for loss are best suited to specific tasks: for example, variants of mean squared error work well for regression problems, whereas variants of cross-entropy loss work well for classification.

Because the loss function takes the output of a neural network as an input, and that neural network output is a composite function comprising many nested activation functions of individual neurons, differentiating the loss function entails differentiating the entire network. To do so, backpropagation uses the chain rule.
 

"Loss function," "cost function" or "error function?"
It’s worth quickly noting that in some contexts, the terms cost function or error function are used in place of loss function, with “cost” or “error” replacing “loss.”

Though some machine learning literature assigns unique nuance to each term, they’re generally interchangeable.¹ An objective function is a broader term for any such evaluation function that we want to either minimize or maximize. Loss function, cost function or error function refer specifically to terms we want to minimize.

Starting from the final layer, a backward pass differentiates the loss function to compute how each individual parameter of the network contributes to the overall error for a single input.

Returning to our earlier example of the classifier model, we would start with the 5 neurons in the final layer, which we’ll call layer L. The softmax value of each output neuron represents the likelihood, out of 1, that an input belongs to their category. In a perfectly trained model, the neuron representing the correct classification would have an output value close to 1 and the other neurons would have an output value close to 0.

For now, we’ll focus on the output unit representing the correct prediction, which we’ll call L_c. L_c’s activation function is a composite function, containing the many nested activation functions of the entire neural network from the input layer to the output layer. Minimizing the loss function would entail making adjustments throughout the network that bring the output of L_c’s activation function closer to 1.

To do so, we’ll need to know how any change in previous layers will change L_c’s own output. In other words, we’ll need to find the partial derivatives of L_c’s activation function.

The output of L_c’s activation function depends on the contributions that it receives from neurons in the penultimate layer, which we’ll call layer L-1. One way to change L_c’s output is to change the weights between the neurons in L-1 and L_c. By calculating the partial derivative of each L-1 weight with respect to the other weights, we can see how increasing or decreasing any of them will bring the output of L_c closer to (or further away from) 1.

But that’s not the only way to change L_c’s output. The contributions L_c receives from L-1 neurons are determined not just by the weights applied to L-1’s output values, but by the actual (pre-weight) output values themselves. The L-1 neurons’ output values, in turn, are influenced by weights applied to inputs they receive from L-2. So we can differentiate the activation functions in L-1 to find the partial derivatives of the weights applied to L-2’s contributions. These partial derivatives show us how any change to an L-2 weight will affect the outputs in L-1, which would subsequently affect the output value of L_c and thereby affect the loss function.

By that same logic, we could also influence the output values that L-1 neurons receive from L-2 neurons by adjusting the contributions that L-2 neurons receive from neurons in L-3. So we find the partial derivatives in L-3, and so on, recursively repeating this process until we’ve reached the input layer. When we’re done, we have the gradient of the loss function: a vector of its partial derivative for each weight and bias parameter in the network.

We’ve now completed a forward pass and backward pass for a single training example. However, our goal is to train the model to generalize well to new inputs. To do so requires training on a large number of samples that reflect the diversity and range of inputs the model will be tasked with making predictions on post-training.

Now that we have the gradients of the loss function with respect to each weight and bias parameter in the network, we can minimize the loss function—and thus optimize the model—by using gradient descent to update the model parameters.

Moving down—descending—the gradient of the loss function will decrease the loss. Since the gradient we calculated during backpropagation contains the partial derivatives for every model parameter, we know which direction to “step” each of our parameters to reduce loss.

Each step reflects the model “learning” from its training data. Our goal is to iteratively update weights until we have reached the minimum gradient. The object of gradient descent algorithms is to find the specific parameter adjustments that will move us down the gradient most efficiently.

Learning rate
The size of each step is a tunable hyperparameter, called the learning rate. Choosing the right learning rate is important for efficient and effective training.

Recall that the activation functions in a neural network are nonlinear. Some gradients may be approximately U-shaped: stepping in one direction moves down the gradient, but continuing to step in that direction will eventually move up the gradient.

A low learning rate ensures we always step in the right direction, but calculating so many changes is time-consuming and computationally expensive. A high learning rate is computationally efficient, but risks overshooting the minimum.
 

Batch size
Another consideration in gradient descent is how often to update weights. One option is to compute the gradients for every example in the training data set, then take an average of those gradients and use it to update parameters. The process is repeated iteratively in a series of training epochs until the error rate stabilizes. This method is batch gradient descent.

When the training data set is very large—as it typically is in deep learning—batch gradient descent entails prohibitively long processing times. Calculating gradients for millions of examples for each iteration of weight updates becomes inefficient. In stochastic gradient descent (SGD), each epoch uses a single training example for each step. While loss might fluctuate on an epoch-to-epoch basis, it quickly converges to the minimum throughout many updates.

Mini-batch gradient descent represents a middle-ground approach. Training examples are randomly sampled in batches of fixed size, and their gradients are then calculated and averaged together. This mitigates the memory storage requirements of batch gradient descent while also reducing the relative instability of SGD.