Learning That ClicksEpisode 4 ／ How does a nature that can't differentiate still descend the slope?

Reading evolution, learning, and consciousness through a single gradient

Evolution Is Descending a Slope Without Ever Differentiating Nature cannot differentiate the loss. And yet life has been descending the slope all along. The secret: "try random guesses, then infer the downhill direction from the correlation between what worked and what didn't" ── Evolution Strategies (ES). It assembles a gradient without ever once using differentiation.

What you'll need: Episode 1's \(\theta\leftarrow\theta-\eta\nabla L\), the idea of an average (expectation), and a mental picture of the normal distribution Key of this episode: \(\hat g=\dfrac{1}{N\sigma^2}\sum_i L(\theta+\varepsilon_i)\,\varepsilon_i\)

Last time (Episode 3), we saw natural selection as the act of climbing the gradient of a fitness landscape. But there was a crack in it that we ought to admit honestly ── nature cannot differentiate. No deity takes the partial derivatives of a loss function and hands out \(\nabla L\) to everyone. DNA knows nothing of the equations of physics, and nobody tells you what shape the slope of the environment even is. The update rule we set up in Episode 1, \(\theta\leftarrow\theta-\eta\nabla L\), assumed that \(\nabla L\) was in hand. So how did a nature that cannot compute a gradient manage to descend the slope? This time we build the answer ── Evolution Strategies (ES) ── with our own hands. What it does is almost anticlimactically plain: take random one-step pokes in every direction, then drift, on average, toward whichever directions worked out. That alone makes \(\nabla L\) appear, without differentiation ever being used. It's not a magic trick. By the end, we'll properly verify that this is an unbiased estimate of the gradient of a smoothed objective function.

01The dilemma of a nature that can't differentiate

Let's recall Episode 1's update rule once more.

Gradient descent (Episode 1, the spine of the series)
$$\theta_{t+1}=\theta_t-\eta\,\nabla L(\theta_t)$$

For this formula to run, you need the gradient \(\nabla L(\theta)\) at your current location \(\theta\). For a neural network we had a device that computes this automatically ── backpropagation (BP, Episode 2). BP works because the network is a smooth composite function, and the chain rule lets you trace the derivative back one layer at a time.

But out in nature there are countless situations where that premise simply doesn't hold. Fitness (the degree to which you survive and leave offspring) is not a smooth function of the input DNA that you can write down as a formula. Whether you get spotted by a predator, whether you find food, whether you fall ill ── all of it comes out the far side of the vast black box we call the environment, and the only thing that gets decided is whether a given individual lived or died. Forget differentiating the insides; you don't even know the formula. This is nature's dilemma.

Black-box optimization, a much wider problem This isn't only about nature. When you don't know the insides of \(L(\theta)\) and feeding in \(\theta\) returns only a value ── minimizing against such a "box" is called black-box optimization. The output of a simulator, the readings from an experiment, the score in a game, or a non-differentiable metric (alive/dead, won/lost). Against these opponents, where a gradient is unobtainable in principle, ES can descend the slope with the very same toolkit. The star of this episode is "life," but its reach is far broader.

Let's boil the problem down to a single question. We can evaluate \(L(\theta)\) (measure its value), but we cannot differentiate it. Under these conditions, how do we find the downhill direction?

02Try random guesses ── scatter perturbations ε

If differentiation is off the table, let's just do it the plain way. Around the current \(\theta\), take several small random steps. We call this "random step" a perturbation \(\varepsilon\), and we draw it from a normal distribution with mean 0 and variance \(\sigma^2\).

How to scatter perturbations (Gaussian distribution)
$$\varepsilon\sim\mathcal N(0,\sigma^2 I),\qquad \theta_i=\theta+\varepsilon_i\quad(i=1,\dots,N)$$

Draw \(N\) perturbations \(\varepsilon_1,\dots,\varepsilon_N\), and make \(N\) individuals \(\theta_i=\theta+\varepsilon_i\), each nudged a little differently. Here \(\sigma\) is "how boldly you nudge" = the strength of the mutation. To liken it to biology: \(\theta\) is the parent's genotype, \(\varepsilon_i\) the mutations, and \(\theta_i\) the children carrying those mutations. Up to here, no differentiation has been used at all. All we used was random numbers.

Next, we evaluate each individual. We measure \(L(\theta_i)=L(\theta+\varepsilon_i)\). Since this is just putting it in the box and getting a value back, no differentiation is needed. In biology, this is the evaluation of "did that child manage to survive" = fitness (with the sign flipped). The smaller \(L\), the better the individual. What we now hold is nothing but data: \(N\) pairs of ⟨the direction stepped, \(\varepsilon_i\); the resulting badness, \(L(\theta+\varepsilon_i)\)⟩. Can we assemble a downhill direction out of this material?

03Estimating direction from the correlation of hits and misses

The intuition goes like this. Head in the opposite direction from any \(\varepsilon_i\) that produced a bad result (large \(L\)). Conversely, you want to move toward directions that produced good results (small \(L\)). In other words, if you take each perturbation \(\varepsilon_i\) and average them weighted by its resulting \(L(\theta+\varepsilon_i)\), the slope of the terrain should surface. Cast into a formula, that is the ES gradient estimator.

The ES gradient estimator ── the star of this episode
$$\hat g\;=\;\frac{1}{N\sigma^2}\sum_{i=1}^{N} L(\theta+\varepsilon_i)\,\varepsilon_i$$

Read aloud ── "take each perturbation \(\varepsilon_i\), weight it by its badness \(L(\theta+\varepsilon_i)\), sum them up, then divide by the head count \(N\) and the variance \(\sigma^2\)." Directions \(\varepsilon_i\) that gave large \(L\) are added with large weight, so \(\hat g\) becomes a vector pointing toward "the bad way (uphill)". That's why the update goes the other way:

The ES update rule ── never differentiates L, not even once
$$\theta\;\leftarrow\;\theta-\eta\,\hat g$$

Notice how closely this resembles Episode 1's \(\theta\leftarrow\theta-\eta\nabla L\) in form. The one and only difference ── in the seat where \(\nabla L\) used to sit, there now sits \(\hat g\), assembled from the correlation between evaluations and perturbations. All ES uses is the correlation between the evaluations \(L(\theta+\varepsilon_i)\) and the perturbations \(\varepsilon_i\). Nowhere is there any operation that differentiates \(L\).

Why divide by \(\sigma^2\) ── the trick of the score function This \(\hat g\) isn't handed down from on high; it comes out of the log-derivative trick (the score-function method ── the same skeleton as REINFORCE in reinforcement learning). Because the probability density \(p_\sigma(\varepsilon)\) of the perturbation \(\varepsilon\) is Gaussian, the gradient of its logarithm (the score) is \(\nabla_\varepsilon \log p_\sigma = -\varepsilon/\sigma^2\) ── and that is where the \(\sigma^2\) appears. The plain intuition of a "weighted average," once written in the language of probability, demands exactly this coefficient. The rigorous derivation (the proof of unbiasedness) is previewed in the next section and delivered in full in Episode 6 using Stein's lemma.

04This is not "finite differences" ── the gradient of a smoothed J

Here I'll head off the point that gets misunderstood most often. "So in effect you're nudging here and there and taking differences, which means you're just approximating \(\nabla L\) by finite differences (numerical differentiation), right?" ── this is wrong. What ES estimates without bias is not the gradient of the raw \(L\), but the gradient of a different function \(J\) that is \(L\) blurred (smoothed).

The smoothed objective function (also the star of Episodes 6 & 7)
$$J(\theta)\;=\;\mathbb{E}_{\varepsilon\sim\mathcal N(0,\sigma^2 I)}\big[\,L(\theta+\varepsilon)\,\big]$$

\(J(\theta)\) is "the value of \(L\) smoothed out by applying a Gaussian of width \(\sigma\) around \(\theta\)" ── frosted glass laid over the terrain, so to speak. The ES estimator \(\hat g\) is an unbiased estimator of the true gradient \(\nabla J(\theta)\) of this \(J\). That is, though it wobbles from one draw to the next depending on the luck of the sample, averaged out it matches \(\nabla J\) exactly (zero bias).

The heart of ES ── unbiasedness (proof in Episode 6)
$$\mathbb{E}\big[\,\hat g\,\big]\;=\;\nabla J(\theta)\qquad(\text{not an approximation, but an equality})$$

"Approximation" and "unbiased estimate" look alike but are utterly different. Finite differences is an approximation that matches \(\nabla L\) only as the step width goes to 0, and it carries a bias that never quite vanishes along with numerical instability from the division. ES, by contrast, need not send the step to 0 ── indeed, \(\sigma\) stays finite, and it hits the gradient of the smoothed function \(J\) that \(\sigma\) determines, dead on, without bias. The wobble (variance) shrinks as you increase the sample size \(N\), but the aim itself (the expectation) is accurate from the start. This viewpoint ── the gradient of \(J\) ── becomes the lead role, unchanged, in Episode 6 (the coincidence of ES and BP) and Episode 7 (blurring the terrain to escape local minima).

05Let's run it ── the population sniffs out the slope's direction

Below are the contours of the 2D loss landscape \(L(x,y)=\tfrac12(0.4\,x^2+1.2\,y^2)+0.3\,x\). The white dot at the center is the current \(\theta\). Around it, \(N\) perturbation points \(\varepsilon_i\) (Gaussian, variance \(\sigma^2\)) are scattered, and each point is colored by its loss \(L\) (green = good (low) / red = bad (high)). The two arrows are the estimated gradient \(\hat g\) that plain ES assembled, and, for the answer key, the true gradient \(\nabla L\) (which in reality nature never gets its hands on). Try moving the population size \(N\) and the mutation strength \(\sigma\).

Figure: A 2D loss landscape (contours) and the ES estimate \(\hat g\) assembled from the population \(\{\varepsilon_i\}\). The arrows point in the gradient (uphill) direction. As \(N\) grows, \(\hat g\) drifts toward the true gradient
good individual (small L) bad individual (large L) ES estimate ĝ true gradient ∇L current θ

Try it and two things come into view. First ── when \(N\) is small, \(\hat g\) points somewhere off from the true gradient and wavers, the arrow jumping every time you press (high variance). Turn \(N\) up and \(\hat g\) slides smoothly into alignment with the true gradient \(\nabla L\). This is the visible form of the unbiasedness that "matches as \(N\to\infty\)" (rigorously in Episode 6). Second ── make \(\sigma\) large and the individuals scatter farther, making the big picture of rough terrain easier to grasp, while \(\hat g\) comes to point not at the "raw \(\nabla L\)" but at "the gradient of a blurrier \(J\)." \(\sigma\) is the knob that sets both the breadth of your field of view and how blurred the terrain you aim at is, at the same time.

06Antithetic sampling lowers the variance

ES's weak point is the high variance we just saw. With few samples the arrow thrashes about. In practice there are many tricks to tame this, but the most basic and beautiful is antithetic sampling ── using perturbations in pairs of \(+\varepsilon_i\) and \(-\varepsilon_i\).

The antithetic-sampling estimator (smaller variance)
$$\hat g\;=\;\frac{1}{2N\sigma^2}\sum_{i=1}^{N}\big(L(\theta+\varepsilon_i)-L(\theta-\varepsilon_i)\big)\,\varepsilon_i$$

Why does it work? The one-sided \(\hat g\) carries a "pedestal" (the baseline offset of \(L\), plus distortion) that has nothing to do with the terrain's slope and becomes noise. Measure both \(+\varepsilon_i\) and \(-\varepsilon_i\) and take the difference, and this symmetric pedestal cancels out, leaving cleanly only the component that bears on the slope. And unbiasedness is preserved all the while. In the biological metaphor: give a certain mutation and its exact opposite mutation to two children, and decide which way to drift by looking only at the difference in the two children's fitness ── something like that. Let's confirm it with concrete numbers.

Let's try it ── hitting the gradient with antithetic sampling

1D L(θ)=θ², current θ=2, σ=1. Estimate using just one pair ε=+1 (and −1)

$$\hat g=\frac{1}{2N\sigma^2}\big(L(\theta+\varepsilon)-L(\theta-\varepsilon)\big)\varepsilon =\frac{1}{2\cdot 1\cdot 1}\big(L(3)-L(1)\big)\cdot 1$$

Substituting L(3)=9, L(1)=1

$$\hat g=\frac{9-1}{2}=4$$

Answer key: the true gradient is ∇L(θ)=2θ, so

$$\nabla L(2)=2\cdot 2=4\qquad\Longrightarrow\qquad \hat g=\nabla L(2)=4\ \ (\text{a match})$$

With just one pair of perturbations, we nailed the true gradient \(4\) exactly. An honest note: the reason this matches so cleanly is that \(L(\theta)=\theta^2\) is a quadratic. Even after smoothing, \(J(\theta)=\theta^2+\sigma^2\) (only a constant shifts), and the gradient is \(\nabla J=2\theta\), unchanged from the raw \(\nabla L\). So in this case alone, the blurred gradient and the raw gradient come out to the same value. On general terrain (with cubic or higher curvature), \(\nabla J\neq\nabla L\), and what ES hits is, after all, the gradient of the blurred \(J\) ── not confusing the two is the key to correctly understanding "this is not finite differences" from Section 4.

◇ ◇ ◇

07The verdict ── what ES can and cannot do

As always, let's judge the tool's power and its limits honestly. Both the appeal of ES and its price are perfectly clear.

QuestionES's answerVerdict
Can it descend the slope without differentiation? Yes. It builds \(\hat g\) from nothing but the correlation between evaluations \(L(\theta+\varepsilon_i)\) and perturbations \(\varepsilon_i\) Yes
Does it work even for a black-box or non-differentiable \(L\)? Yes. Even if the insides can't be written as a formula, all you need is to measure the value Yes
Is it "just approximating \(\nabla L\) by finite differences"? No. It's an unbiased estimate of the gradient of the smoothed objective \(J\), not an approximation (proven in Episode 6) That reading is ✗
Is it sample-efficient? Unbiased but high-variance. It demands many samples (= many trials, or in biology, many deaths) Efficiency is so-so
The honest bottom line ── unbiased, but costly

The ES estimate \(\hat g\) is unbiased (the aim is accurate), but its variance is large and its sample efficiency poor. To get a decent direction you must take a large population size \(N\), and out in nature that means an enormous number of deaths. That evolution is slow and cruel is precisely the price of this high variance. In exchange, ES demands no differentiability whatsoever, and works on black boxes and on discontinuous metrics alike ── it has a universality BP cannot imitate.

And the biggest caution ── the received wisdom that "ES is nothing but a crude approximation of BP (backpropagation)" is false. What ES hits is not an approximation of the raw \(\nabla L\), but the gradient of the smoothed \(J\) itself. So what, exactly, is the relationship between differentiating BP and non-differentiating ES? Next time (Episode 5) we'll prepare the mathematics (nonstandard analysis) for counting rigorously ── not just by feel ── the "infinite sample" of \(N\to\infty\). Then, in Episode 6, through Stein's lemma, we'll show that ── ES and BP coincide, mathematically and exactly, in a certain limit. Today's \(\hat g\) is the first step toward that coincidence.

Practice problems (solvable with this episode's formulas alone)
  1. 1D \(L(\theta)=\theta^2\), current \(\theta=3\), \(\sigma=1\). Using antithetic sampling with one pair \(\varepsilon=+1\) (and \(-1\)), find \(\hat g\). Does it match the true gradient \(\nabla L(3)\)?
    See the answer
    \(\hat g=\dfrac{1}{2\cdot1\cdot1}\big(L(4)-L(2)\big)\cdot1=\dfrac{16-4}{2}=6\). The true gradient is \(\nabla L(3)=2\cdot3=6\), so it matches. Since \(L=\theta^2\) is quadratic, the gradient of the blurred \(J\) also stays \(2\theta\), unchanged, so it hits cleanly.
  2. Can we say the ES estimator \(\hat g=\dfrac{1}{N\sigma^2}\sum_i L(\theta+\varepsilon_i)\,\varepsilon_i\) is an unbiased estimate of the gradient \(\nabla L\) of the raw \(L\)? If not, what is it an unbiased estimate of?
    See the answer
    No. What \(\hat g\) estimates without bias is not the raw \(\nabla L\), but the gradient \(\nabla J\) of the smoothed objective function \(J(\theta)=\mathbb E_\varepsilon[L(\theta+\varepsilon)]\). That is, \(\mathbb E[\hat g]=\nabla J(\theta)\). The two coincide only in special cases such as \(L\) being a quadratic (because \(J\) is \(L\) blurred by \(\sigma\)).
  3. When you double the population size \(N\), which properties of the estimate \(\hat g\) change, and how? Answer separately for bias (the offset of the expectation) and variance (the wobble).
    See the answer
    The bias does not change ── \(\hat g\) is unbiased to begin with (\(\mathbb E[\hat g]=\nabla J\)), and increasing \(N\) leaves the expectation at \(\nabla J\). What changes is the variance: since it's the average of independent samples, the variance drops roughly in proportion to \(1/N\). So doubling \(N\) roughly halves the variance, the arrow's wavering shrinks, and it drifts toward the true direction. The aim was accurate from the start; only the precision improves.

Episode 4 summaryEvolution is descending a slope without differentiating

Nature cannot differentiate the loss ── because fitness is a non-differentiable metric passed through a black box (STEP 01). So ES scatters Gaussian perturbations \(\varepsilon_i\) around \(\theta\) (STEP 02), takes a weighted average of each perturbation by its evaluation \(L(\theta+\varepsilon_i)\) to assemble the gradient \(\hat g=\tfrac{1}{N\sigma^2}\sum_i L(\theta+\varepsilon_i)\varepsilon_i\), and descends via \(\theta\leftarrow\theta-\eta\hat g\) (STEP 03). All it uses is the correlation between evaluations and perturbations; there is zero differentiation of \(L\). This \(\hat g\) is not a finite-difference approximation, but an unbiased estimate of the gradient of the smoothed objective \(J(\theta)=\mathbb E_\varepsilon[L(\theta+\varepsilon)]\) (STEP 04). The price of unbiasedness, though, is high variance, and antithetic sampling \(\pm\varepsilon_i\) is the basic trick for lowering it (STEP 06). The form is identical to Episode 1's \(\theta\leftarrow\theta-\eta\nabla L\) ── it's just that \(\hat g\) has taken \(\nabla L\)'s seat.

Differentiating BP and non-differentiating ES. We're finally drawing close to the claim previewed in Episode 1 that "the two are the same computation." To get there, next time we first obtain a tool for handling the infinity of \(N\to\infty\) rigorously, not by feel.

This document is Episode 4 of the "Learning That Clicks" series, a piece of reading for high-schoolers and undergraduates interested in physics, math, and AI. The Evolution Strategies (ES) gradient estimator \(\hat g=\tfrac{1}{N\sigma^2}\sum_i L(\theta+\varepsilon_i)\varepsilon_i\), and the fact that it is an unbiased estimator (\(\mathbb E[\hat g]=\nabla J\)) of the gradient \(\nabla J\) of the smoothed objective \(J(\theta)=\mathbb E_{\varepsilon\sim\mathcal N(0,\sigma^2 I)}[L(\theta+\varepsilon)]\), is an established result derived from the score-function method (the log-derivative trick, isomorphic to REINFORCE in reinforcement learning); the rigorous proof is given in Episode 6 using Stein's lemma. Antithetic sampling \(\hat g=\tfrac{1}{2N\sigma^2}\sum_i(L(\theta+\varepsilon_i)-L(\theta-\varepsilon_i))\varepsilon_i\) is a standard variance-reduction technique and preserves unbiasedness. The worked example here (\(\hat g=\nabla L\) for \(L=\theta^2\)) matches cleanly only because \(L\) is a quadratic, making \(\nabla J=\nabla L\), a special case; in general \(\nabla J\neq\nabla L\) (what ES hits is the gradient of the smoothed \(J\)), as made explicit in the body. The figure's terrain \(L(x,y)=\tfrac12(0.4x^2+1.2y^2)+0.3x\) is an illustrative model, and the coloring and arrows are drawn by actually computing \(\hat g\) from the samples at each moment. The random numbers are standard pseudo-random numbers (Gaussian generation via the Box–Muller method), and a fresh sample is drawn each time you press a button or operate a slider. Nonstandard analysis (Episode 5), the coincidence of ES and BP (Episode 6), and the easing of local minima by smoothing the terrain (Episode 7) are covered in later episodes. ── To print, use your browser's "Print" and "Save as PDF" (in the print version the sliders and answers are static and hidden).

Print / save as PDF: ⌘+P (Ctrl+P on Windows). On screen, move "population size N" and "mutation strength σ," and press "Resample" to scatter the individuals anew with fresh random numbers. Turn N up and watch the ES-estimate arrow drift toward the true gradient. Click "See the answer" to open a solution.