Learning That ClicksEpisode 2 ／ Backpropagation ── the ledger that builds the gradient automatically

Evolution, learning, and consciousness, read through a single gradient

Backpropagation Is
the Ledger of the Chain Rule In Episode 1 we treated the gradient \(\nabla L\) as something already in hand. How does a neural network compute it automatically?
The answer is the derivative of a composite function ── the chain rule ── run backward, from output to input, by a diligent bookkeeper.

Tools you'll need: Episode 1's gradient descent, the derivative of a composite function (the chain rule) Key to this episode: \(\dfrac{dL}{d\theta}=\dfrac{dL}{dh}\dfrac{dh}{d\theta}\)

Last time, we distilled learning into a single equation ── \(\theta\leftarrow\theta-\eta\,\nabla L\). Nudge the parameters a little in the direction opposite the gradient. Walk downhill. But that equation smuggled in a quiet assumption: the gradient \(\nabla L\) is already in hand. So who computes that \(\nabla L\)? In a neural network with a hundred million weights, how do you obtain, for all hundred million, "how much does the loss change if I move each weight"? Done naively, this computation is hopelessly expensive ── yet there is a trick that dispatches the whole thing for roughly the same cost as a single forward pass. That is backpropagation (backprop, BP). Its true nature is no magic: it is nothing but the derivative of a composite function you learned in high school ── the chain rule ── carried backward from output to input by a meticulous bookkeeper. Today we open that ledger one page at a time and watch, with our own hands, the moment \(\nabla L\) is born.

01Last episode's homework ── where does ∇L come from?

To run the update rule \(\theta\leftarrow\theta-\eta\,\nabla L\), you need every component of the gradient \(\nabla L=\big(\partial L/\partial\theta_1,\dots,\partial L/\partial\theta_n\big)\). The most naive approach is to do exactly what the definition says: poke each knob a little. Move parameter \(\theta_i\) by \(\varepsilon\) and divide by the change in the loss ── the so-called finite difference.

The naive method (finite difference) ── correct, but expensive
$$\frac{\partial L}{\partial \theta_i}\;\approx\;\frac{L(\theta+\varepsilon e_i)-L(\theta)}{\varepsilon}$$

This equation is correct. But to obtain one \(\partial L/\partial\theta_i\), you have to compute \(L(\theta+\varepsilon e_i)\), the loss with only \(\theta_i\) shifted ── that is, you must run the entire network once (one forward pass). The gradient has \(n\) components. So naively that means \(n\) forward passes. With a million weights, every single step downhill costs a million runs of the network. Learning would never finish. Backpropagation folds those \(n\) passes into essentially one. The key is the chain rule that follows.

02The chain rule ── the derivative of a composite function is a chain of multiplications

A neural network is a composite function, simple functions stacked many layers deep. "Multiply by a weight," "apply a nonlinearity," "multiply by another weight"… stacked up, and at the end it produces a loss \(L\). The derivative of a composite function obeys just one rule ── the chain rule. If the loss is a two-layer stack \(L=g(h(\theta))\), then

The chain rule (derivative of a composite function)
$$\frac{dL}{d\theta}\;=\;\frac{dL}{dh}\cdot\frac{dh}{d\theta}$$

Take "how much \(h\) moves when \(\theta\) moves (\(dh/d\theta\))" and multiply it by "how much \(L\) moves when \(h\) moves (\(dL/dh\))." You multiply together the sensitivities along the way. It works the same when there are more layers: for \(L=f_3(f_2(f_1(\theta)))\), you just add one factor at a time ──

Even with many layers, it's just a chain of products
$$\frac{dL}{d\theta}\;=\;\frac{dL}{df_2}\cdot\frac{df_2}{df_1}\cdot\frac{df_1}{d\theta}$$

Here lies an asymmetry that pays off later. This product gives the same answer whichever end you start multiplying from, but the order of multiplication changes the amount of computation. Multiplying from the input side is the forward direction; multiplying from the output side (the \(L\) side) is backpropagation. Why going backward is cheaper is a bill we settle in Section 6. First, let's run this chain on a concrete two-layer network.

03The forward pass ── first, push values forward

Consider the smallest possible network: one input, two weights. Input \(x\), weights \(w_1,w_2\), target \(y\). Along the way we insert one sigmoid, \(\sigma(z)=1/(1+e^{-z})\). Compute the values in order, from left to right ── this is the forward pass.

Forward pass ── values flowing forward, from input to loss
$$z_1=w_1 x,\quad a_1=\sigma(z_1),\quad \hat y=w_2 a_1,\quad L=\tfrac12(\hat y-y)^2$$

First mix linearly to get \(z_1\), bend it with the nonlinearity to get \(a_1\), multiply by a weight again for the prediction \(\hat y\), then square the gap from the target to get the loss \(L\). Here is the crucial part ── in the forward pass we keep not only the answer \(L\) but all the intermediate values \(z_1,\,a_1,\,\hat y\) that came out along the way. These entries become the master ledger for the bookkeeping we call backpropagation. Let's record one thing in advance ── the derivative of the sigmoid. It has a very well-behaved form.

The sigmoid's derivative can be written in terms of itself Differentiating \(\sigma(z)=\dfrac{1}{1+e^{-z}}\) gives \(\sigma'(z)=\sigma(z)\big(1-\sigma(z)\big)\). In other words, as long as you have \(a_1=\sigma(z_1)\), already computed in the forward pass, the derivative \(\sigma'(z_1)=a_1(1-a_1)\) comes out in a single multiplication. You don't even need to recompute \(e^{-z}\) at \(z_1\) ── another dividend of "keeping the intermediate values on file."

04The backward pass ── the ledger that distributes the error δ backward

Once the forward pass reaches \(L\), we now trace the chain rule in reverse, starting from the output side. Begin with "how much \(L\) moves when the prediction \(\hat y\) moves," then hand that sensitivity backward one layer at a time. The quantity being handed along is traditionally called \(\delta\) (delta) ── the "share of blame for the loss" assigned to each node.

Backward pass ── the chain rule from the back. Distribute δ and settle each gradient
$$\delta_{\text{out}}=\frac{\partial L}{\partial \hat y}=\hat y-y$$ $$\frac{\partial L}{\partial w_2}=\delta_{\text{out}}\cdot a_1,\qquad \frac{\partial L}{\partial a_1}=\delta_{\text{out}}\cdot w_2$$ $$\frac{\partial L}{\partial z_1}=\frac{\partial L}{\partial a_1}\cdot \sigma'(z_1),\qquad \frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial z_1}\cdot x$$

Each line is one link of the chain rule. Starting from the output's blame \(\delta_{\text{out}}=\hat y-y\) (exactly the amount you missed by), the right-hand term flows into the left. Since \(\hat y=w_2 a_1\), the sensitivity to \(w_2\) is \(a_1\) times, and the sensitivity to \(a_1\) is \(w_2\) times. Next, because we pass through \(a_1=\sigma(z_1)\), we multiply by \(\sigma'(z_1)\) to get the blame for \(z_1\); finally, since \(z_1=w_1 x\), we multiply by \(x\) to get the gradient for \(w_1\). Can you see that the \(\delta\) once sent backward is reused at each layer? This is the beauty of the "ledger": \(\partial L/\partial a_1\) is reused directly in computing \(\partial L/\partial z_1\), and \(\partial L/\partial z_1\) in turn flows into \(\partial L/\partial w_1\). Let's settle the accounts all the way through with actual numbers.

Let's try it ── computing ∇L to the end on a 2-layer net

Setup: x=1, y=0, w₁=0.5, w₂=0.8 (using σ(0.5)=0.6225)

① Forward pass (values forward)

$$z_1=w_1 x=0.5,\quad a_1=\sigma(0.5)=0.6225,\quad \hat y=w_2 a_1=0.8\times0.6225=0.4980$$ $$L=\tfrac12(\hat y-y)^2=\tfrac12(0.4980)^2=0.1240$$

② Backward pass (error δ backward)

$$\delta_{\text{out}}=\hat y-y=0.4980$$ $$\frac{\partial L}{\partial w_2}=\delta_{\text{out}}\cdot a_1=0.4980\times0.6225=0.3100$$ $$\frac{\partial L}{\partial a_1}=\delta_{\text{out}}\cdot w_2=0.4980\times0.8=0.3984$$ $$\sigma'(z_1)=\sigma(0.5)\big(1-\sigma(0.5)\big)=0.6225\times0.3775=0.2350$$ $$\frac{\partial L}{\partial z_1}=\frac{\partial L}{\partial a_1}\cdot\sigma'(z_1)=0.3984\times0.2350=0.0936$$ $$\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial z_1}\cdot x=0.0936\times1=0.0936$$

With just one round trip of backpropagation, we have the gradient for every parameter, \(\nabla L=\big(\partial L/\partial w_1,\ \partial L/\partial w_2\big)=(0.0936,\ 0.3100)\). All that's left is to plug it into Episode 1's update rule \(w\leftarrow w-\eta\,\nabla L\). The weight with the larger blame, \(w_2\), moves more ── the weight that swung the loss harder gets corrected harder.

05Try it ── flow δ through the computational graph

Below is the network above, laid out as a single computational graph. The top row (amber) is the forward pass ── values flowing left to right; the bottom row (green) is the backward pass ── gradients \(\delta\) flowing right to left. The boxed \(\partial L/\partial w_1,\ \partial L/\partial w_2\) are the gradients you use directly for the update. \(x=1,\ y=0\) are fixed; move the knobs for the weights \(w_1,\ w_2\) and each value of the forward pass and each gradient of the backward pass reassemble simultaneously.

Figure: computational graph of the 2-layer net. Top = forward pass (values forward) / bottom = backward pass (gradients δ backward). Move w₁, w₂ and ∇L is recomputed instantly. The readout shows "how L changes if you take one step down this gradient."
Forward pass (values forward) Backward pass (gradients δ backward) Gradient used for the update ∂L/∂w

Move things around a bit and you start to see the ledger breathe. Since we're aiming for \(y=0\), the closer \(\hat y\) is to \(0\), the smaller both \(L\) and the gradients become ── set \(w_2\) near \(0\) and \(\hat y\to0\), and the loss all but vanishes. Conversely, make \(w_1\) large and \(a_1\) gets pushed up onto the flat shoulder of the sigmoid (\(\sigma'\approx0\)), so \(\partial L/\partial w_1\) withers away ── this is the seed of the "vanishing gradient" we'll meet in a later episode. Whichever weight you touch, confirm that the bottom row's gradients are all updated at once, in a single backward flow.

06Why it's fast ── the O(n)-vs-O(1) bookkeeping trick

Here we return to Section 1's homework. Finite differences cost one forward pass per gradient component, \(n\) passes in all ── a weight of \(O(n)\), directly proportional to the number of parameters. Backpropagation, however, runs one forward pass and one backward pass and spits out all \(n\) gradients together. However many parameters there are, the number of round trips does not change ── measured against a single forward pass, the extra cost is essentially \(O(1)\).

Cost of getting the same ∇L (in units of one forward pass)
$$\text{Finite difference:}\ \underbrace{O(n)\ \text{passes}}_{\text{proportional to # of parameters}}\qquad\text{vs}\qquad \text{Backprop:}\ \underbrace{O(1)}_{\text{all gradients in one round trip}}$$

The reveal is the direction in which you multiply the chain rule (the asymmetry from Section 2). Compute the shared output-side factor \(\delta\) once, and reuse it at every layer. Once you've found \(\partial L/\partial z_1\), you can pass it straight into \(\partial L/\partial w_1\) ── had you multiplied forward, from the input side, you'd have to rebuild the sensitivity chain from scratch for every parameter, sliding right back to \(O(n)\). It is precisely because you flow backward that the intermediate \(\delta\) becomes shared property across all parameters. This is the heart of reverse-mode automatic differentiation, and backpropagation is simply its name in machine learning.

What the "ledger" metaphor really is In the forward pass you enter the intermediate values (\(z_1,a_1,\hat y\)) into the master ledger; in the backward pass you consult it while transcribing and settling each node's share of blame \(\delta\), in order from the output ── it is accounting itself. That's why "no number, once computed, is ever built again." The meticulousness of allocating each portion of the loss to each weight, down to the last cent, is what gives rise to the \(O(1)\) speed.
◇ ◇ ◇

07The verdict ── what backpropagation can and cannot do

Even a powerful bookkeeper has ground rules that must hold for it to work. Let's judge honestly.

QuestionBackpropagation's answerVerdict
Can it produce the gradient \(\nabla L\) exactly, and cheaply? Yes. Unlike the finite-difference approximation, it gives the exact value via the chain rule, for all parameters, in one round trip = \(O(1)\) Can do
Does it work for any loss and any network? It requires \(L\) to be differentiable and the intermediate values of the forward pass to be retained (memory and smoothness are prerequisites) Requires prerequisites
Can it be used for evolution in nature? No. Living things have no god to differentiate their loss ── the very machinery for flowing a chain of derivatives backward simply doesn't exist Cannot
The honest line ── beyond the world where you can differentiate

Backpropagation, as reverse-mode automatic differentiation, is an established technique that delivers exact gradients cheaply. Nearly all of today's deep learning stands on top of it. But let's honestly draw two lines ── (1) it requires differentiability and retention of the computational graph. It only runs when \(L\) is smooth and you can keep the intermediate values of the forward pass on file. (2) Biological evolution has none of this. There is no device in nature to differentiate DNA, and no master ledger to record intermediate values.

So in a world where you can't differentiate, how do you walk downhill? The answer is a method that descends without differentiating ── evolution strategies (ES) (Episode 4). And before that, we'll confirm that this loss landscape and biology's fitness landscape were the same picture (Episode 3). More astonishingly still, "BP that differentiates" and "ES that doesn't" turn out, in Episode 6, to be mathematically the same computation. The ledger we assembled today will, at that point, step back onto this stage as one of the two leads in the comparison.

Practice problems (solvable with just this episode's equations)
  1. On the same 2-layer net, let \(x=1,\ y=0,\ w_1=1,\ w_2=1\). Using \(\sigma(1)=0.7311\), find \(L,\ \delta_{\text{out}},\ \partial L/\partial w_2\).
    See the answer
    \(z_1=1,\ a_1=\sigma(1)=0.7311,\ \hat y=1\times0.7311=0.7311\). \(L=\tfrac12(0.7311)^2=0.2673\). \(\delta_{\text{out}}=\hat y-y=0.7311\). \(\partial L/\partial w_2=\delta_{\text{out}}\cdot a_1=0.7311\times0.7311=0.5345\).
  2. In backpropagation, why does \(\partial L/\partial z_1=\partial L/\partial a_1\cdot\sigma'(z_1)\) pick up a \(\sigma'(z_1)\) along the way? Explain in the language of the chain rule.
    See the answer
    Because \(z_1\) is a composite that affects \(L\) only through \(a_1=\sigma(z_1)\). By the chain rule \(\dfrac{\partial L}{\partial z_1}=\dfrac{\partial L}{\partial a_1}\dfrac{\partial a_1}{\partial z_1}\), the local sensitivity \(\partial a_1/\partial z_1=\sigma'(z_1)\) enters as the factor for that one link. This is why, on the sigmoid's shoulder (\(\sigma'\approx0\)), the gradient nearly vanishes.
  3. On a net with \(n=1{,}000{,}000\) parameters, roughly how many forward passes' worth of effort does it take to get the gradient \(\nabla L\) once, for finite differences and for backpropagation respectively?
    See the answer
    Finite differences take one forward pass per component, about \(1{,}000{,}001\) passes including the base point = \(O(n)\). Backpropagation is one forward pass + one backward pass, roughly two forward passes' worth = \(O(1)\), independent of the number of parameters. A million-fold difference. This is exactly why learning becomes practical.

Episode 2 summary∇L is born from a backward-running ledger

The \(\nabla L\) demanded by the update rule \(\theta\leftarrow\theta-\eta\nabla L\) can be approximated by naive finite differences in \(O(n)\) forward passes (STEP 01). Because a neural network is a composite function, its derivative chains into a product via the chain rule (STEP 02). The forward pass pushes values and intermediate values forward and records them (STEP 03); the backward pass distributes the error \(\delta\) backward from the output, starting from \(\delta_{\text{out}}=\hat y-y\), settling the gradient of each layer (STEP 04; in the worked example, \(\nabla L=(0.0936,\,0.3100)\)). By multiplying the chain from the back and reusing the intermediate \(\delta\), all gradients are obtained exactly, in one round trip = \(O(1)\) ── this is the heart of reverse-mode automatic differentiation (STEP 06). But it presupposes differentiability and retention of intermediate values, and cannot be used for evolution (STEP 07).

To the \(\nabla L\) that was Episode 1's "assumption," we've now properly connected a supply. Next time we step away from the machine for a moment and confirm that this loss landscape is the same picture as biology's fitness landscape. And then, on to a way of descending that landscape without differentiating ── where, in Episode 6, we'll be reunited with today's ledger.

This document is Episode 2 of the "Learning That Clicks" series, a reading piece for high-schoolers and undergraduates interested in physics, mathematics, and AI. Backpropagation applies the chain rule of composite functions \(\frac{dL}{d\theta}=\frac{dL}{dh}\frac{dh}{d\theta}\) in reverse mode, delivering the exact value of the gradient \(\nabla L\) for all parameters together in one forward pass + one backward pass (\(O(1)\) in units of one forward pass) ── this is an established mathematical and computational technique. It is also a known fact that gradient approximation by finite differences requires \(O(n)\) forward passes for \(n\) parameters, and that the sigmoid \(\sigma(z)=1/(1+e^{-z})\) satisfies \(\sigma'(z)=\sigma(z)(1-\sigma(z))\). The 2-layer net in this piece is a minimal model for exposition, and values such as \(\sigma(0.5)=0.6225\) are rounded to about the fourth decimal place as ballpark figures (\(\nabla L\approx(0.0936,\,0.3100)\)). That backpropagation presupposes differentiability and retention of the computational graph, that biological evolution has neither, and the mathematical equivalence with evolution strategies (ES) (Episode 6) are treated in later episodes. ── To print, use your browser's "Print" and "Save as PDF" (in the print version, the sliders and answers are frozen and hidden).

Print / save as PDF: ⌘+P (Ctrl+P on Windows). On screen, move the weights "w₁" and "w₂" and the forward-pass values and backward-pass gradient ∇L reassemble at once, with "how L changes if you step downhill" shown in the readout. Click "See the answer" to open a solution.