Learning That ClicksEpisode 6 ／ Unification ── ES = BP, the heart of the series

Reading evolution, learning, and consciousness through a single gradient

Evolution and Learning Were the Same Computation "Evolution strategies (ES)," which descend a slope without differentiating, and "backpropagation (BP)," which descend by differentiating. Two things that look utterly different turn out — through a single equation, Stein's lemma — to be the exact same gradient.

Tools you'll need: Gaussian integrals, integration by parts, and Episode 5's nonstandard analysis Key of the episode: $\dfrac{1}{\sigma^2}\,\mathbb{E}\!\left[L(\theta+\varepsilon)\,\varepsilon\right]=\nabla L_\sigma(\theta)$

Over the series so far, we've picked up two ways of "going downhill." One is Episode 2's backpropagation (BP) ── differentiate the loss, compute the gradient $\nabla L$ straight, and descend: a sharp, local tool. The other is Episode 4's evolution strategies (ES) ── use no differentiation at all; scatter many perturbations and average the "directions that turned out good": a blunt but robust, population-level tool. Learning that differentiates and evolution that doesn't. In philosophy the two look diametrically opposed. And yet, in truth, the two are computing exactly the same gradient. ES is BP wearing the face of "integration," and BP is ES wearing the face of "differentiation." What proves this coincidence is a classic of statistics ── Stein's lemma. This episode is the heart of the series. Skipping not a single step of the derivation, let's walk on our own two feet all the way to the final identity.

01Two ways down ── integration and differentiation

First, let's set the two side by side. Let's borrow ahead of time the smoothed loss (Gaussian smoothing) that will be the star of Episode 7. It's the loss $L$ "leveled out" by blurring it around the center $\theta$ with Gaussian noise $\varepsilon$ of standard deviation $\sigma$.

Smoothed loss (blur around θ with σ)

$$J(\theta)\;=\;\mathbb{E}_{\varepsilon\sim\mathcal{N}(0,\sigma^2 I)}\!\bigl[\,L(\theta+\varepsilon)\,\bigr]\;=\;L_\sigma(\theta)$$

Lowering this $L_\sigma$ is the shared goal of the two. So how do the two get their hands on its gradient $\nabla L_\sigma(\theta)$?

ES (the evolution, integration face). Scatter $N$ perturbations $\varepsilon_1,\dots,\varepsilon_N$ from the center $\theta$ using a Gaussian, and measure only the evaluation value $L(\theta+\varepsilon_i)$ of each. No differentiation. Then take "the average of the perturbations weighted by their evaluation values" ── this is ES's gradient estimate. A global operation that gathers the values of the loss and integrates (averages) them.

Set the two estimators side by side

ES: average of (evaluation value × perturbation) (no differentiation, global)

$$\hat{g}_{\mathrm{ES}}\;=\;\frac{1}{\sigma^2}\cdot\frac{1}{N}\sum_{i=1}^{N} L(\theta+\varepsilon_i)\,\varepsilon_i$$

BP: average of gradients in a neighborhood (with differentiation, local)

$$\hat{g}_{\mathrm{BP}}\;=\;\frac{1}{N}\sum_{i=1}^{N}\nabla L(\theta+\varepsilon_i)\quad\bigl(\text{as}\ N\to\infty,\ \mathbb{E}[\nabla L(\theta+\varepsilon)]\bigr)$$

BP (the learning, differentiation face). At each point, differentiate the loss and read the gradient $\nabla L$ directly. For the gradient of the blurred loss, just average the gradients in the neighborhood. A local operation that gathers the slopes of the loss.

One is an integral of values, the other a differential of slopes. The information used and the computational philosophy differ. And yet these two converge to the same quantity. The bridge is Stein's lemma.

02Stein's lemma ── integration by parts for the Gaussian

The key is one beautiful equation the Gaussian distribution possesses. The average of a quantity "multiplied" by Gaussian noise turns into the average of that quantity "differentiated" ── a theorem that bridges integration (averaging) and differentiation, made exactly for this episode.

Stein's lemma (integration by parts for the Gaussian)

$$\varepsilon\sim\mathcal{N}(0,\sigma^2 I),\quad \mathbb{E}\!\left[\,\varepsilon\, f(\varepsilon)\,\right]\;=\;\sigma^2\,\mathbb{E}\!\left[\,\nabla f(\varepsilon)\,\right]$$

($f$ is taken to be a sufficiently nice function ── differentiable, and not blowing up as $\varepsilon$ heads off to infinity.)

The left side is the average of "the product of the noise $\varepsilon$ and the function value $f$"; the right side is the average of "the derivative of $f$." Left is an integral of values, right is an average of slopes. This is exactly the skeleton of the ES–BP relationship. Why does this equation hold ── it comes out in one line from a single property of the Gaussian density. In the next section let's prove the one-dimensional case all the way to the end (the multidimensional case is just applying the same argument to each component).

03Proof in one dimension ── integration by parts to the finish

The heart of the proof is a differential equation satisfied by the Gaussian density $p(\varepsilon)$. Differentiate the Gaussian and it comes back as itself, multiplied by $-\varepsilon/\sigma^2$ ── from this single point, integration by parts does everything else for us.

Let's try it ── integration by parts for the Gaussian

① The Gaussian density, and its derivative

$$p(\varepsilon)=\frac{1}{\sqrt{2\pi}\,\sigma}\,e^{-\varepsilon^2/(2\sigma^2)} \qquad\Longrightarrow\qquad p'(\varepsilon)=-\frac{\varepsilon}{\sigma^2}\,p(\varepsilon)$$

② So "ε × density" can be rewritten as "the derivative of the density"

$$\varepsilon\,p(\varepsilon)\;=\;-\,\sigma^2\,p'(\varepsilon)$$

③ Substitute this into the average $\mathbb{E}[\varepsilon f(\varepsilon)]$

$$\mathbb{E}\!\left[\varepsilon f(\varepsilon)\right] =\int_{-\infty}^{\infty}\!\varepsilon\, f(\varepsilon)\,p(\varepsilon)\,d\varepsilon =-\,\sigma^2\!\int_{-\infty}^{\infty}\! f(\varepsilon)\,p'(\varepsilon)\,d\varepsilon$$

④ Integrate by parts (the boundary term vanishes as $p\to 0$)

$$\int f\,p'\,d\varepsilon =\underbrace{\bigl[\,f(\varepsilon)\,p(\varepsilon)\,\bigr]_{-\infty}^{\infty}}_{=\,0} -\int f'(\varepsilon)\,p(\varepsilon)\,d\varepsilon =-\int f'(\varepsilon)\,p(\varepsilon)\,d\varepsilon$$

⑤ Put it back into ③, and the two minus signs line up

$$\mathbb{E}\!\left[\varepsilon f(\varepsilon)\right] =-\,\sigma^2\left(-\int f'(\varepsilon)\,p(\varepsilon)\,d\varepsilon\right) =\sigma^2\!\int f'(\varepsilon)\,p(\varepsilon)\,d\varepsilon =\sigma^2\,\mathbb{E}\!\left[f'(\varepsilon)\right]$$

And so $\mathbb{E}[\varepsilon f(\varepsilon)]=\sigma^2\,\mathbb{E}[f'(\varepsilon)]$ ── Stein's lemma (one-dimensional) is rigorously proved. No approximation whatsoever. Just the Gaussian density's property $p'=-(\varepsilon/\sigma^2)p$ and integration by parts. The boundary term vanishes because, as long as $f$ is gentler than the Gaussian's tail, $f(\varepsilon)p(\varepsilon)\to 0$.

Why "integration by parts" works so well here Integration by parts is the operation of "shoving the differentiation onto whatever it's multiplying." Here, we take the differentiation that was on the $f$ we want to measure, temporarily let the Gaussian density $p$ shoulder it (③), translate that into "multiply by $\varepsilon$" via $p'=-(\varepsilon/\sigma^2)p$, and use integration by parts to hand it back to the $f$ side once more (④⑤). On the Gaussian, differentiation and "multiplying by $\varepsilon$" are two sides of the same coin ── this is the true identity of Stein's lemma, and it is the very reason ES (multiplication) and BP (differentiation) later coincide.

04The central identity ── ES = BP = ∇Lσ

The preparations are complete. Substitute $f(\varepsilon)=L(\theta+\varepsilon)$ into Stein's lemma. From the chain rule, note that the gradient with respect to $\varepsilon$ equals the gradient with respect to $\theta$ ── $\nabla_\varepsilon L(\theta+\varepsilon)=\nabla_\theta L(\theta+\varepsilon)$.

Let's try it ── applying it to the main problem

① Put $f(\varepsilon)=L(\theta+\varepsilon)$ into Stein's lemma

$$\mathbb{E}\!\left[\varepsilon\,L(\theta+\varepsilon)\right]\;=\;\sigma^2\,\mathbb{E}\!\left[\nabla L(\theta+\varepsilon)\right]$$

② Divide both sides by $\sigma^2$

$$\frac{1}{\sigma^2}\,\mathbb{E}\!\left[L(\theta+\varepsilon)\,\varepsilon\right]\;=\;\mathbb{E}\!\left[\nabla L(\theta+\varepsilon)\right]$$

③ Swap the averaging and differentiation ($\nabla_\theta$ can come outside the average over $\varepsilon$)

$$\mathbb{E}\!\left[\nabla_\theta L(\theta+\varepsilon)\right]\;=\;\nabla_\theta\,\mathbb{E}\!\left[L(\theta+\varepsilon)\right]\;=\;\nabla L_\sigma(\theta)$$

Stringing the three together on one line, the backbone identity of this series appears.

The central identity ── ES and BP are the same gradient

$$\underbrace{\frac{1}{\sigma^2}\,\mathbb{E}\!\left[\,L(\theta+\varepsilon)\,\varepsilon\,\right]}_{\text{ES: integral of values, global}} \;=\; \underbrace{\mathbb{E}\!\left[\,\nabla L(\theta+\varepsilon)\,\right]}_{\text{BP: average of slopes, local}} \;=\; \nabla\,\mathbb{E}\!\left[L(\theta+\varepsilon)\right] \;=\; \nabla L_\sigma(\theta)$$

The far left is ES ── a global "integral" quantity, using no differentiation, that averages the evaluation values $L(\theta+\varepsilon)$ weighted by the perturbations $\varepsilon$. The middle is BP ── a local "differential" quantity that averages the gradients obtained by differentiating at infinitely many neighboring points. These two are exactly equal. And moreover both are nothing but the gradient $\nabla L_\sigma$ of the smoothed loss $L_\sigma$.

ES is BP on the smoothed loss.
Evolution and learning were computing the same gradient.

It's not an approximation. It's not a finite difference. Stein's lemma, an exact equation, ties the two worlds together with a single line. Evolution that doesn't differentiate and learning that does were merely computing the same gradient of the same terrain $L_\sigma$, just wearing different faces.

05Let's run it ── the theorem shakes hands before your eyes

The figure below is a visualization of the identity itself. On top of a two-dimensional loss terrain (contour lines) we place a single point $\theta$ (red), and from there scatter $N$ Gaussian perturbations (amber dots). The green arrow is the ES estimate $\hat{g}$, built from the left-side formula using only the evaluation values of the scattered points. The black arrow is the true smoothed gradient $\nabla L_\sigma$ (the right side), which can be computed analytically for this terrain. Turn $N$ up ── the ES arrow converges toward the true gradient as if drawn to it (the estimate's scatter is $\propto 1/N$). Change $\sigma$ and the strength of the blur ── i.e., the true gradient itself ── changes.

Figure: The central identity on a loss terrain (sum of negative Gaussian wells). Green = ES estimate ĝ = (1/Nσ²)Σ L(θ+εᵢ)εᵢ (values only, no differentiation) / black = true smoothed gradient ∇Lσ (right side, analytic value). Raise N and green converges to black ── this is the theorem's evidence itself.

Population size N (number of perturbations)40 Blur σ (spread of perturbations)0.35

True smoothed gradient ∇Lσ (right side) ES estimate ĝ (left side, values only) Center θ Perturbation θ+εᵢ

What about $N=2$? The green arrow points in a wildly different direction each time, and its length is all over the place too ── no wonder, since we're trying to guess a gradient from just two evaluation values. But raise $N$ to a few hundred and green lands exactly on black. Just by gathering evaluation values and averaging, without differentiating, the gradient you'd get by differentiating is recovered. The readout's "angle difference" heads toward $0^\circ$, and the "magnitude ratio" toward $1$. This is the visible meaning of "ES is an unbiased estimate of BP."

06Integration and differentiation shake hands ── in the language of nonstandard analysis

Using the language of nonstandard analysis from last time (Episode 5), this convergence becomes even more transparent. Rather than making the population size $N$ large while keeping it finite, let's take $N$ to be hyperfinite (an infinite hypernatural number). Then the ES sum becomes a hyperfinite sum.

The nonstandard reading ── the standard part recovers the gradient

$$\mathrm{st}\!\left(\frac{1}{\sigma^2}\cdot\frac{1}{N}\sum_{i=1}^{N} L(\theta+\varepsilon_i)\,\varepsilon_i\right) \;=\;\frac{1}{\sigma^2}\,\mathbb{E}\!\left[L(\theta+\varepsilon)\,\varepsilon\right] \;=\;\nabla L_\sigma(\theta)$$

The evaluation-value-weighted average over a population of infinitely many individuals. Its standard part $\mathrm{st}$ (the "visible value" with the infinitesimal fluctuation shaved off) matches exactly the right-side expectation ── the true gradient. As we saw in Episode 5, each individual of the population is an infinitesimal perturbation filling the monad (infinitesimal neighborhood) of the center $\theta$, and the consensus of those infinitesimals recovers the gradient of the standard world. For finite $N$, a fluctuation of $\propto 1/N$ (the deviation from the infinitesimal, in Episode 5's terms) remains in this consensus; but if $N$ is hyperfinite, that deviation is infinitesimal and $\mathrm{st}$ erases it.

Integration and differentiation shake hands on the Gaussian Ordinarily integration (gathering values and averaging) and differentiation (the slope at a single point) are opposite operations. Yet in Stein's lemma they are joined by an equals sign. The left side is "a global quantity integrating the whole population," the right side "a local quantity, the gradient at a single point." The symmetry of the distribution called the Gaussian reconciles this global and this local. The same truth that evolution discovered "in the macro of a population" and learning discovered "in the micro of a gradient," each separately ── that truth was $\nabla L_\sigma$.

◇ ◇ ◇

07The verdict ── what is exact, and what to watch out for

This coincidence is powerful, but exactly what it claims needs to be judged carefully. Let's put three common turns of phrase, all three, on trial.

Claim	Examination	Verdict
"ES is a finite-difference approximation of BP (just mimicking the derivative with a difference quotient)."	No. ES isn't numerically differentiating the raw $L$; it's an unbiased estimate of the gradient of the smoothed loss $L_\sigma$. Not a finite difference, but a rigorous average quantity given by Stein's identity.	✗ (rejected)
"ES and BP coincide exactly on the raw loss $L$."	What coincides is the gradient of the σ-blurred $L_\sigma$, not the gradient of the raw $L$. The two coincide only in the limit $\sigma\to 0$ (and when $L$ is smooth).	handle with care
"Stein's identity is exact (not an approximation)."	Correct. Under mild assumptions of integrability and smoothness, it's a complete equation derived from integration by parts. In Section 3 we proved it using no approximation at all.	survives

The honest line ── what is "the same" is the gradient of Lσ, not the raw L

The identity $\tfrac{1}{\sigma^2}\mathbb{E}[L(\theta+\varepsilon)\,\varepsilon]=\nabla L_\sigma(\theta)$ is an exact equation from Stein's lemma (under mild assumptions of integrability and smoothness). But the sense in which ES and BP are "the same" here is only about the gradient of the smoothed loss $L_\sigma$. As long as $\sigma$ is finite, $\nabla L_\sigma$ deviates from the gradient $\nabla L$ of the raw loss ── the true nature of this deviation, and the role $\sigma$ plays, is precisely next time's subject.

Even so, the claim is plenty strong. Evolution (ES) and learning (BP) are not separate principles but two faces (integration and differentiation) of one and the same gradient computation. One gathers the population's values, the other differentiates at a single point ── the apparatus is completely different, yet the gradient they point to is the same. This is the backbone of this series. Next time we'll reread this $\sigma$ as "resolution / temperature," and watch how blurring smooths the terrain and fills in local minima.

Practice problems (solvable with just this episode's identity)

In Stein's lemma, set $f(\varepsilon)=\varepsilon$ (one-dimensional, $\varepsilon\sim\mathcal N(0,\sigma^2)$). What comes out? Compute the left and right sides separately.
See the answer
Left side $\mathbb{E}[\varepsilon\cdot\varepsilon]=\mathbb{E}[\varepsilon^2]=\sigma^2$. Right side $\sigma^2\,\mathbb{E}[f'(\varepsilon)]=\sigma^2\,\mathbb{E}[1]=\sigma^2$. Both sides come out $\sigma^2$ and coincide, showing that Stein's lemma contains the familiar fact "the variance is $\sigma^2$."
When the loss is the quadratic $L(\theta)=\tfrac12\|\theta\|^2$, find the gradient of the smoothed loss $\nabla L_\sigma(\theta)$. How does it differ from the raw gradient $\nabla L=\theta$?
See the answer
$L_\sigma(\theta)=\mathbb{E}[\tfrac12\|\theta+\varepsilon\|^2]=\tfrac12\|\theta\|^2+\tfrac12\,\mathbb{E}\|\varepsilon\|^2=\tfrac12\|\theta\|^2+\text{constant}$. The constant vanishes under differentiation, so $\nabla L_\sigma(\theta)=\theta=\nabla L(\theta)$. For a quadratic, smoothing doesn't change the gradient (so this is the special case that looks like "coincides with the raw $L$"). For a general nonlinear $L$, a deviation appears ── that's the verdict table's "handle with care."
For the ES estimate $\hat g=\tfrac{1}{\sigma^2}\cdot\tfrac1N\sum_i L(\theta+\varepsilon_i)\varepsilon_i$, if you quadruple $N$, by roughly what factor does the estimate's "scatter (standard deviation)" change? Can you confirm it in the figure?
See the answer
The variance is $\propto 1/N$, so quadrupling $N$ makes the variance $1/4$, and the standard deviation, its square root, $1/2$. In the figure, as you enlarge $N$ the green arrow's range of thrashing shrinks by exactly half each time, and you can see it converging to the black arrow. This is the typical behavior of an "unbiased but variance-carrying estimate."

Episode 6 summaryEvolution and learning are two faces of one and the same gradient

ES is a global tool that only integrates evaluation values; BP is a local tool that differentiates to get the gradient (STEP 01). What bridges the two is Stein's lemma $\mathbb{E}[\varepsilon f(\varepsilon)]=\sigma^2\mathbb{E}[\nabla f]$ (STEP 02), which can be proved rigorously from just the Gaussian density's $p'=-(\varepsilon/\sigma^2)p$ and integration by parts (STEP 03). Putting in $f(\varepsilon)=L(\theta+\varepsilon)$ and tidying up gives the central identity $\tfrac1{\sigma^2}\mathbb{E}[L(\theta+\varepsilon)\varepsilon]=\mathbb{E}[\nabla L(\theta+\varepsilon)]=\nabla L_\sigma(\theta)$ ── ES = BP = the gradient of the smoothed loss (STEP 04). In the figure, the higher $N$ goes the more the ES estimate converges to the true gradient (STEP 05), and in nonstandard analysis the standard part of the hyperfinite sum recovers the gradient (STEP 06). But what is "the same" is the gradient of $L_\sigma$, not the raw $L$ (STEP 07).

Evolution that doesn't differentiate and learning that does were not different things. They were merely computing the same gradient of the same terrain $L_\sigma$, through the two faces of integration and differentiation ── this coincidence is the heart of this series. Only one question remains. So then, what is $\sigma$?

Coming up next ── Episode 7 Blur it, and the terrain becomes smooth. This time, the smoothed loss $L_\sigma$ tied ES and BP together. Its $\sigma$ is not merely "the size of the noise." Blur the terrain with $\sigma$ and the fine valleys (local minima) fill in, leaving only the large flow ── $\sigma$ is resolution and temperature. Grasp the big picture with a large $\sigma$, then shrink it to descend into the details. We take on local minima, our nemesis ever since Episode 1, with the weapon of blurring.

This document is Episode 6 of the "Learning That Clicks" series, a reading for high-school and university students interested in physics, math, and AI. Stein's lemma (for $\varepsilon\sim\mathcal N(0,\sigma^2 I)$, $\mathbb{E}[\varepsilon f(\varepsilon)]=\sigma^2\mathbb{E}[\nabla f(\varepsilon)]$) is a classical statistical equation due to C. Stein, and the one-dimensional proof here (the Gaussian density's relation $p'=-(\varepsilon/\sigma^2)p$, integration by parts, and the vanishing of the boundary term) holds rigorously under the mild assumptions of integrability, sufficient smoothness, and decay at the tails. Applying this to $f(\varepsilon)=L(\theta+\varepsilon)$ and using the exchange of averaging and gradient (justified under standard conditions such as dominated convergence) yields the identity $\tfrac1{\sigma^2}\mathbb{E}[L(\theta+\varepsilon)\varepsilon]=\mathbb{E}[\nabla L(\theta+\varepsilon)]=\nabla L_\sigma(\theta)$, which shows that the evolution-strategies (ES) gradient estimate is an unbiased estimate of the gradient of the smoothed loss $L_\sigma$ (the same relation as the Evolution Strategies formulation of Salimans et al.). To emphasize: this coincidence is not about the gradient of the raw loss $L$, but strictly about the gradient of the σ-smoothed $L_\sigma$. The terrain in this document's figure is an illustrative model built from a sum of negative Gaussian functions, and the true $\nabla L_\sigma$ is drawn exactly using the property that Gaussian smoothing closes analytically (the convolution of two Gaussians is a Gaussian). That the estimator's variance decreases as $\propto 1/N$ is a general property of the sample mean of independent samples. The interpretation via the standard part of a hyperfinite sum in nonstandard analysis is a rereading connected to Episode 5's framework. ── To print, use your browser's "Print" and choose "Save as PDF" (in the print version the sliders and answers are static and hidden).

Print / PDF: ⌘+P (Ctrl+P on Windows). On screen, raising "Population size N" makes the green ES-estimate arrow converge to the black true gradient ∇Lσ. Use "Blur σ" to change the strength of the smoothing, and "Redraw" to redraw the perturbations. "See the answer" opens each solution.