Settling the debt from Episode 1: "which variables to keep has an element of human choice"
Episode 1's "honest line" said: "which variables to keep is not handed to you automatically by physical law ── there is an element of human choice depending on the question." This bonus settles that debt. Information theory in fact comes with a full set of theorems and formulas about discarding. "Processing cannot increase information" is a theorem ── the data processing inequality ── and it is the information-side statement that irrelevant things never come back. "Fix the tolerable error and the required number of bits is determined" is rate–distortion theory, and it is what the curve in Bonus ② really was. "Fix the question and the optimal coarse-graining is unique" is the information bottleneck. And at the end, the hardest evidence of all ── an information-theoretic inequality actually proved a theorem of the renormalization group.
Start with the foundation. Coarse-grain \(X\) (micro) into \(Y\), then coarse-grain that into \(Z\) ── write the chain as \(X\to Y\to Z\).
\(I\) is mutual information (how much is shared). You can never learn more from the processed version than from the unprocessed one. Obvious-looking, but it is a theorem. And the renormalization group is precisely a chain of coarse-grainings, so information about the microscopic decreases monotonically at every step. Episode 4 said irrelevant directions shrink away; this inequality is what guarantees they never come back.
Conversely, coarse-graining can be done well or badly precisely because the rate of loss differs. Discard the same number of bits and how much answer-relevant information survives depends on how you chose. So what is the optimal way to discard?
Shannon built two theories of compression. One is lossless. The other assumes from the start that you will lose things.
Fix the tolerable error \(D\) and the required bits \(R\) have a hard lower bound ── a wall you cannot get under. The curve in Bonus ② showing "how many steps b bits buys you" was exactly this \(R(D)\).
For Gaussian sources \(R(D)\) can be worked out by hand, and the answer is startlingly intuitive ── draw one water line.
Let the independent components have variances \(\sigma_1^2,\sigma_2^2,\dots\). Pick a water level \(\theta\):
$$D=\sum_i \min(\theta,\sigma_i^2),\qquad R=\sum_i \max\!\left(0,\ \tfrac12\log_2\frac{\sigma_i^2}{\theta}\right)$$In words ── components below the water line (\(\sigma_i^2<\theta\)) are discarded wholesale (not one bit is spent on them). Only components above the line are recorded, and only for the part sticking out.
Translate into Episode 4 and it is almost word for word: the water level \(\theta\) is the threshold separating relevant from irrelevant. In the renormalization group, "change the scale and small components vanish by themselves"; in information theory, "compress optimally and small components are discarded by themselves." The same operation written from the physics side and from the information side.
Here, for the first time, we can give a quantitative meaning to a phrase this series has kept vague. What state of affairs is "the world has layers"?
Draw the rate–distortion curve \(R(D)\) and ──
| Shape of the curve | Meaning | |
|---|---|---|
| A layered world | a steep drop, then a plateau (a knee) | there is a natural stopping point: "squash this far and you're done" ── a level of description exists |
| A world without layers | smooth, no knee | stopping anywhere costs the same ── no natural level of description exists |
The position of the knee is the height of that world's "floor." You may stop at temperature and pressure because there is a knee there. Without a knee, coarse-graining further always costs proportionally more, and "the story closes at this floor" never happens.
On the left of the figure are the variances of the source's 16 components (bars) and the water level \(\theta\) (horizontal line). On the right is the resulting rate–distortion curve and where you currently are.
Raise and lower the water level with the slider. Components submerged below it are discarded (grey); only those above receive bits. Switch the source with the button and ──
This figure ties together the vocabulary of Episodes 4 and 6 ── above the line = relevant, below = irrelevant, the knee = a layer, no knee = coarse-graining broken (Episode 5's chaos and turbulence are exactly the knee-less worlds).
There is still a gap, though. We are still the ones deciding how to measure "error." Without specifying what the compression is for, the optimal way to discard isn't determined.
So put the question itself into the equation.
\(X\) is the raw data, \(T\) the coarse-grained representation, \(Y\) what you want to know. "Squash \(X\) as hard as possible while keeping the information about \(Y\)" written directly as an optimisation (Tishby–Pereira–Bialek 1999), with \(\beta\) as the greed dial.
The answer to Episode 1's debt: "which variables to keep" is not settled by human taste. Specify 〈what you want to know (\(Y\))〉 and the rest is determined uniquely. The "causal states" of Bonus ② are the special case \(Y=\)"the future."
And in physics, \(Y\) picks itself naturally ── the macroscopically measurable quantities, or the sufficiently distant future. That is where temperature and pressure come from. Coarse-graining is not arbitrary; the moment you fix the question, the world decides.
Not "information theory and the renormalization group resemble each other" but one of them proved the other. This is the hardest fact in this bonus.
In 1986 Zamolodchikov showed, for two-dimensional field theories, that there exists a quantity \(c\) that decreases monotonically along the renormalization group flow (the c-theorem). Roughly, \(c\) counts the theory's degrees of freedom. So flowing from UV (fine) to IR (coarse) necessarily reduces the degrees of freedom ── the rigorous field-theory form of Episode 4's "what you discard doesn't come back."
Now the point. In 2004 Casini and Huerta proved this c-theorem from somewhere else entirely ── from the strong subadditivity of entanglement entropy, a purely information-theoretic inequality. Proofs along the same lines have since been given for the three-dimensional F-theorem and the four-dimensional a-theorem.
an information-theoretic inequality (strong subadditivity)
↓
renormalization-group monotonicity theorems (c-, F-, a-theorems)
Not "similar," not "there's a correspondence." A theorem of physics was derived starting from an inequality about information. The data processing inequality saying "coarse-graining loses information" and the c-theorem saying "renormalization loses degrees of freedom" turn out to be one and the same monotonicity.
Finally, let's give a formal shape to the closing line of Bonus ②.
When explaining data, choose the model that minimises "bits to write down the model" + "bits to write down the deviations from it" (Rissanen). Too simple and the second term explodes; too complex and the first does. Occam's razor becomes an accounting of bits.
Kepler's laws are the compression of a vast table of Martian observations into the very short description "ellipse" ── a law is a compression, and being compressible is the same as being coarse-grainable. That is what Bonus ② meant by "in a universe that cannot be coarse-grained, no law can be written."
Established: the data processing inequality; rate–distortion theory and the exact reverse water-filling solution for independent Gaussian sources under squared error (Shannon, Berger); the information bottleneck formulation (Tishby–Pereira–Bialek 1999) and its optimality conditions; that computational mechanics' causal states are the minimal sufficient representation for prediction; Zamolodchikov's c-theorem (1986); Casini–Huerta's proof of the c-theorem from strong subadditivity of entanglement entropy (2004) and of the three-dimensional F-theorem (2012), together with proofs of the four-dimensional a-theorem along the same lines; and Rissanen's minimum description length principle ── all established results.
This series' phrasing, not theorems: (1) "Having layers means having a knee in the R–D curve" is this series' formulation. I think it is suggestive and useful, but it is not a standard definition (nor does "knee" have a unique mathematical definition). (2) The information bottleneck has known weaknesses ── it degenerates for deterministic maps, the choice of \(\beta\) remains, and its application to deep learning (Tishby et al.'s "compression phase") is still disputed on reproducibility grounds. "Fix the question and it's unique" holds once \(Y\) and the error measure are fixed. (3) Casini–Huerta's proof works under the standard assumptions of field theory (Lorentz invariance, unitarity, etc.). (4) Kolmogorov complexity is uncomputable; MDL is its practical stand-in. (5) The figure uses an idealised independent Gaussian source, not the spectrum of any real physical system.
"Coarse-graining loses information" is the data processing inequality, guaranteeing that irrelevant things never return. "How far may you squash" has the lower bound \(R(D)\), solved exactly for independent Gaussian sources by reverse water-filling ── discard everything below the water level \(\theta\) wholesale and record only what is above. That water level is precisely Episode 4's relevant / irrelevant threshold.
And the heart of this bonus ── having layers means having a knee in the rate–distortion curve. The knee is the natural stopping point that says "you may stop at this floor," and knee-less worlds (Episode 5's chaos and turbulence) never grow a natural level of description. Episode 1's debt ── "which variables to keep is a human choice" ── is answered by the information bottleneck: specify what you want to know and the coarse-graining is unique. The only freedom left to us is the choice of question.
The hardest evidence: Casini and Huerta proved the renormalization group's c-theorem from strong subadditivity of entanglement entropy, a purely information-theoretic inequality. "Coarse-graining loses information" and "renormalization loses degrees of freedom" were one and the same monotonicity.
Print / make a PDF: ⌘+P (Ctrl+P on Windows). On screen, moving the slider changes the water level, the discarded components and the required bits. The button switches sources so you can compare knee against no knee. "See the answer" opens each solution.