Reading evolution, learning, and consciousness through a single gradient
In Episode 4 we saw that evolution is a way to descend the slope without differentiating (evolution strategies, ES). In Episode 9 we saw that this computation comes with a price tag called energy. Today we join the two. Write down both the ES update of evolution and the learning of a brain, and you get exactly the same single line ── \(\theta\leftarrow\theta-\eta\,\hat g\). So what makes them different? Only two things. The time scale over which the gradient is measured (a generation, or a single moment within one lifetime), and who pays the cost of measuring that gradient (death, or experience). To measure a single gradient, evolution needs many deaths and a long time. But if a brain measures the gradient while alive, the same computation can be done without dying, orders of magnitude more cheaply. Learning is a "shortcut" that brings the gradient computation evolution had been doing endlessly forward into life itself. In this episode we walk the ground on foot to confirm how, in the equations, that shortcut is the same, and where and by how much it is cheaper.
Recall the ES gradient from Episode 4. Even if the loss \(L(\theta)\) cannot be differentiated, we could estimate the gradient by adding a small jitter \(\varepsilon_i\) (noise) to the parameters \(\theta\) and judging the result as good or bad.
Here, to measure a single term \(L(\theta+\sigma\varepsilon_i)\) means letting a whole individual with a slightly tweaked genome live, and observing how it does over its lifetime. If it does well, it leaves offspring; if it does poorly ── it leaves none and vanishes. So one sample = one individual's lifetime = for the most part, one death. And the sum has \(N\) terms. To gain a single gradient \(\hat g_{\text{evo}}\) worth one step, you need \(N\) evaluations, that is, \(N\) generations' worth of time, and that many deaths. In the language of Episode 9, the cost per sample is "one lifetime's worth of energy" ── this is what it means for the sample efficiency to be at its worst.
But what if a given individual could measure the gradient within its own lifetime, without dying? Creatures with a brain do exactly this. They adjust the brain's weights \(w\) little by little, and with each experience \(x_i\) compute the gradient from "the gap between prediction and reality" ── this is what backpropagation (BP, Episode 2) and reinforcement learning are made of.
Here, to measure a single term \(\nabla_w\ell(w;x_i)\) means one individual learns from one experience while staying alive. Fail, and it does not die ── it stumbles, it hurts, and next time it does better. It earns \(N\) samples by reusing a single life. So even for the same sum of \(N\) terms, the cost it pays is experience, not death. In the bits/joule of Episode 9: by packing thousands of samples into a single life, it can gather gradient information orders of magnitude more cheaply. This is the reversal of sample efficiency.
The important point is that both \(\hat g_{\text{evo}}\) and \(\hat g_{\text{learn}}\) are estimates of the same "gradient of the loss." Only the way the estimate is built (brute-force sampling, or differentiation) and the way it is paid for differ; the direction they point is the downhill direction of the same slope.
Write the estimated gradient as \(\hat g\), and both evolution and learning fit neatly into the next single line.
Let's set the two implementations side by side in this equation.
| Evolution (ES) | Within-lifetime learning (brain) | |
|---|---|---|
| Update rule | \(\theta\leftarrow\theta-\eta\,\hat g_{\text{evo}}\) | \(w\leftarrow w-\eta\,\hat g_{\text{learn}}\) |
| One sample | one individual's lifetime | one experience within a lifetime |
| Time for one update | a generation (years~) | a moment within a lifetime (seconds~) |
| Who pays the cost | death | experience |
The differences are essentially two. (1) The time scale over which the gradient is measured (a generation vs. a moment within a lifetime), and (2) who pays the cost (death vs. experience). Everything else is the same computation. So we can say this ── learning is the same gradient computation evolution had been doing, redone cheaply and quickly by replacing death with experience. The true nature of the shortcut is not the invention of a new computation, but a switch in the "way of paying" for the same computation.
A two-tier structure then appears naturally. A living thing has two parameters ── its genome \(\theta_{\text{gene}}\) and its brain's weights \(w\) ── each optimized on a different time scale, but by the same update rule. The slow outer loop is evolution; the fast inner loop is learning.
The brain's initial state \(w_0\) is specified by the genome \(\theta_{\text{gene}}\) (\(w_0=\phi(\theta_{\text{gene}})\)) ── what structure it has, and what it finds easy to learn, is the design you are born with. And what evolution is optimizing is not the direct behavior, but "the good initial state and way of learning" itself. The outer loop selects "the ability to learn well." This is exactly the same nested structure as what AI calls meta-learning (learning to learn). Evolution does not drill in the individual answers; it keeps selecting learners that can reach the answer quickly.
The figure below pits two ways of racing to reach a certain target fitness \(F^*\). The horizontal axis is time/cost (generations), the vertical axis is fitness. The brown staircase is evolution only (a tiny improvement each generation, with a death marker in each generation). Green is an individual that has within-lifetime learning (it approaches the goal while alive). The slider "plasticity" is how much can be learned within one lifetime ── raise it, and you can see a crossover in which green reaches the goal faster and the required deaths dwindle.
At plasticity \(p=0\), green lies exactly on top of brown (an individual that cannot learn has only evolution to rely on). Raise \(p\), and green already leaps to a high fitness in its very first lifetime, and the number of generations = the number of deaths needed to reach the goal drops before your eyes. And once \(p\ge F^*\) ── learning in the very first generation alone reaches the goal, and deaths are nearly zero. The bill evolution had been paying with dozens of generations of death, learning pays off with experience within a single life. This is the effect of the "shortcut."
Suppose a certain gradient estimate needs N=1000 samples. First, count by "death."
$$\text{evolution only}:\ N=1000\ \text{individuals}\times1\text{ life}=1000\ \text{deaths}\;(\approx1000\ \text{generations})$$ $$\text{within-lifetime learning}:\ 1\ \text{individual tries}\ 1000\ \text{times in a lifetime}=\underline{\text{deaths}\,0},\ \text{the estimate is nearly the same}$$Next, multiply by the energy from Episode 9 (the price tag of one sample)
$$E_{\text{evo}}\approx N\cdot E_{\text{life}}=1000\times(2\times10^{11}\,\mathrm{J})=2\times10^{14}\,\mathrm{J}$$ $$E_{\text{learn}}\approx 1\cdot E_{\text{life}}=2\times10^{11}\,\mathrm{J}\quad(\text{1000 trials housed within a single life})$$For the same \(N=1000\) gradient estimate, evolution uses 1000 lives and learning uses 1 life. In energy it is about \(N=10^3\) times cheaper (1000×). In time it is even more extreme: against a generation time \(T_{\text{gen}}\approx20\) years, one trial takes seconds ── so the time ratio to arrival is \(T_{\text{gen}}/t_{\text{trial}}\sim(6\times10^8\,\mathrm{s})/(1\,\mathrm{s})\approx\)a billion times faster. The shortcut works by orders of magnitude, in both cost and time.
The shortcut does something even more fascinating. An advantageous behavior acquired through learning eventually guides evolution toward fixing it genetically. This is called the Baldwin effect (proposed in 1896; demonstrated on a computer by Hinton–Nowlan in 1987).
Here is the mechanism. Suppose a certain good behavior is something that genes alone cannot hit precisely, one that "you only reach by learning." An individual that can learn can approach it during its lifetime and gain high fitness. Then, genes that "are born near that good region and have the ability to learn" become advantageous under selection. Over generations, the genetic initial values themselves are drawn toward the good region, and in the end are fixed toward being able to do it innately, without learning (genetic assimilation).
There is, however, a trap of anthropomorphism here. Evolution did not think, "learning is handy, so let's invent it." It just happened that variants who could learn left more offspring ── that is all. There is no designer who sees ahead. In the next verdict, we draw this line honestly.
| Claim | Substance | Verdict |
|---|---|---|
| Learning is the same update rule as evolution, redone with better sample efficiency | Both are \(\theta\leftarrow\theta-\eta\hat g\). Only the time scale and the way of paying differ. | Correct |
| Learning replaces death with experience and is overwhelmingly cheap (Episode 9) | The same gradient estimate, done with 1 life instead of \(N\) lives. About \(N\)× cheaper in energy, and orders of magnitude cheaper in time. | Correct |
| "Evolution invented learning with a purpose" | There is no foresightful designer. Efficient plasticity was merely selected for as a result (anthropomorphism). | Wrong |
There is solid grounding, in both theory and computer experiment, for the idea that "learning = a cheaper gradient estimate, and is selected for by virtue of that efficiency," and that "learning guides and accelerates evolution (the Baldwin effect / genetic assimilation)." The direction ── learning is an efficiency improvement of the same computation as evolution ── is firm.
But we draw two lines. First, "evolution discovered / invented learning" is nothing but a teleological turn of phrase. There is no foresight; efficient variants simply left more offspring. Second, not all learning is gradient-based ── association, imitation, reward-free exploration, and other kinds of learning are hard to reduce to gradient descent. The "same update rule" of this article is an accurate but not monolithic claim about learning that can be written as a gradient.
And now, one step toward the finale. The peak of efficiency does not stop here. Evolution replaced "real death" with "real experience" to make it cheaper. So then ── what if we could make even "real trials" unnecessary? On a world model inside the head, compute the gradient without actually stumbling. Try it in imagination, fail in imagination. Next time, the finale, we give the name consciousness to that ultimate shortcut.
Evolution's ES gradient \(\hat g_{\text{evo}}\) has one sample = one individual's lifetime = for the most part one death, so a sum of \(N\) terms requires \(N\) generations and \(N\) deaths (STEP 01). The brain's within-lifetime learning \(\hat g_{\text{learn}}\) earns the same \(N\) samples as experience within a single life ── 0 deaths, orders of magnitude cheaper (STEP 02). The two are the same update rule \(\theta\leftarrow\theta-\eta\hat g\); only the time scale and the way of paying differ (STEP 03). So the structure becomes nested ── the outer loop = evolution slowly on the genome, the inner loop = learning fast on the brain. Evolution is a meta-learner that selects "the ability to learn well" (STEP 04). Raise plasticity and the deaths needed to arrive vanish (STEP 05). Learning smooths the terrain and guides and accelerates evolution = the Baldwin effect (STEP 06). But "evolution invented learning" is anthropomorphism that imports foresight; correctly, efficient plasticity was merely selected for as a result (STEP 07).
The true nature of the shortcut was not the invention of a new computation, but a switch in the way of paying for the same gradient computation. Death into experience. And next time, that switch advances by its final step ── experience, too, into imagination.
Print / save as PDF: ⌘+P (on Windows, Ctrl+P). On screen, drag the "plasticity" slider and green (the individual with learning) arrives sooner, while the red "death" markers dwindle. "Watch the animation" replays the race, and "Show the answer" opens each solution.