A History of UNIX That Clicks / Bonus 2Bootstrapping Trust 2

The modern answer to the threat of Bonus 1 ── shrink the seed, and check it with diversity

Bootstrappable Builds ── Shrinking the Seed Trusting Trust (Bonus 1) said ── "you have no choice but to trust a binary you didn't make yourself." The modern answer is twofold ── shrink the seed you trust down to a size a human can actually read by hand (357 bytes); and double-check it with a compiler of a different lineage.

Prerequisites: Bonus 1 (Trusting Trust), Episode 9 (the seed) The hope: trust small, and confirm with diversity

In Bonus 1 we saw Thompson's dark twin ── even if a compiler's source is clean, as long as you build with a descendant of a poisoned binary the Trojan survives, and trust hits bottom at "something you didn't make yourself." So are we helpless? No. Just as the hope of the main series (the self-hosting loop, the hand-carved seed) had a modern answer, this threat, too, has two powerful countermeasures ── 1. shrink the seed down to a human-readable size (Bootstrappable / Full-Source Bootstrap), and 2. double-compile with a compiler of a different lineage to cross-check (DDC). The grand finale of the chicken-and-egg "hope."

01Strategy 1 ── make the seed you trust small enough to read by hand

The heart of the problem in Bonus 1 was that "the first binary you have no choice but to trust is too big to inspect." Then ── just make that seed as small as you possibly can. The Bootstrappable Builds project minimizes the number and size of binaries that must be trusted. And its achievement is remarkable ── from a seed of just 357 bytes, called hex0, it builds up the entire GNU system from source alone (Full-Source Bootstrap).

// hex0 ── a 357-byte seed. It just reads space-separated hex and emits bytes $ cat hello.hex0 7C 00 00 00 # This is just a sequence of hex. A human can follow it by eye $ ./hex0 hello.hex0 > hello # hex → bytes (a minimal assembler) You only have to trust these 357 bytes. Everything else builds up from source

On top of these 357 bytes, slightly larger tools build up, one by one, from source. In the figure below, climb from the seed to gcc, and on to the GNU system. It's a straight descendant of Episode 4's "everything from a small core" and Episode 9's "the bootstrap staircase."

Figure 1: Full-Source Bootstrap. From the 357-byte hex0 seed, each stage ── hex → M2 → Mes → TinyCC → gcc ── builds the next, larger tool from source, climbing all the way to the entire GNU system
Crux 1 this time ── "the hand-carved seed" is reborn in the modern age

In Episode 9 we saw that the first bytes were carved by human hands. Full-Source Bootstrap brings that back into the modern age ── if you shrink the seed you must trust down to 357 bytes, a human can actually read all of it by eye. The "binary you have no choice but to trust, and can't inspect" of Bonus 1 becomes a size that can be audited. The motto "the first egg was carved by hand" comes back to life as a practical matter of security.

◇ ◇ ◇

02Strategy 2 ── double-check with a compiler of a different lineage (DDC)

The other answer goes in a different direction from shrinking the seed ── it's a method to verify a suspect compiler you already have, as-is. David A. Wheeler's Diverse Double-Compiling (DDC). The idea goes like this ── prepare a trusted compiler A of a different lineage from the suspect compiler X, compile the same source twice through it, and compare the result bit-for-bit with X's binary.

Here's the brilliance. Thompson's poison is in neither the source nor the different-lineage A (the poison lurks only in the lineage of X's binary). So the binary rebuilt via A is clean, and ── if X is poisoned, the rebuilt result and X disagree, and the poison is exposed. In the figure below, toggle X between "honest" and "poisoned," and watch the comparison fail only when it's poisoned.

Figure 2: Diverse Double-Compiling. Compile the same source (twice) with the suspect X and with a trusted compiler A of a different lineage, and compare the result bit-for-bit with X's binary. They disagree only when X is poisoned
Crux 2 this time ── bit-for-bit identity as the verdict (Episode 3 returns)

For DDC to hold, the same source must yield the same binary ── reproducible builds. This is exactly the same idea as Episode 3's "verify by the bit-for-bit match of stage2 == stage3." Back then we said "a correct compiler produces the same output no matter what it's built with." DDC combines that with diversity, making it the verdict that "if it matches when rebuilt through a different lineage, the binary is faithful to the source."

03The reveal ── this is how you overcome "you can't make all of it"

The lesson of Bonus 1 was "you can't trust code you didn't completely create yourself." In reality, no one can make all of it. But the two strategies overcome that in practice ── shrink the scope of what you trust down to 357 bytes that a human can read through, and further cross-check with a diverse second compiler. Instead of "make all of it," it's "trust small, and independently confirm each other." The poison Thompson showed to be "undetectable in principle" became, now, detectable in principle.

CONNECTING VOICE ── a triple handshake with the main series This bonus shakes hands with the main series in three ways. Episode 3 (the bit-for-bit match of stage2==stage3 = the ancestor of reproducible builds), Episode 4 (everything from a small core = shrinking the seed), Episode 9 (the hand-carved seed = the 357-byte auditable seed). The toolkit we picked up as we traced the chicken and egg all the way down becomes, as-is, the answer to security ── deeply understanding the structure of bootstrapping was directly connected to protecting it.
THE HONEST LINE ── not a cure-all, but a sure step forward

(1) Even with the 357-byte seed, you're still trusting the kernel and CPU that run that hex0 (the hardware/microcode of Episode 10). Full-Source Bootstrap minimizes the software seed, but hardware trust remains a separate problem. (2) DDC assumes that "the different-lineage A is not independently poisoned (does not collude with X)." How to secure a fully independent A is a practical crux.

(3) Reproducible Builds is a large, ongoing effort, and not all software has reached it yet. (4) Guix's Full-Source Bootstrap, GNU Mes, stage0, and live-bootstrap are developing projects, and their composition and numbers change with the version (357 bytes is a representative example of a hex0-seed). The point is the direction: "make the seed you trust small and verifiable, and confirm it with diversity."

Practice problems (solvable with just this installment)
  1. Which problem from Bonus 1 does Full-Source Bootstrap ease, and how?
    See the answer
    It eases the problem that "the first binary you have no choice but to trust is too big to inspect," by shrinking the seed to 357 bytes so a human can read all of it = making it auditable.
  2. Why can DDC see through Thompson's poison?
    See the answer
    The poison is in neither the source nor the trusted, different-lineage compiler A; it lurks only in the lineage of the suspect X's binary. So the result of re-double-compiling via A is clean, disagrees bit-for-bit with the poisoned X, and is exposed.
  3. What property, the same as in Episode 3, does DDC assume?
    See the answer
    Reproducible builds (that the same source yields a bit-for-bit identical binary). The same idea as Episode 3's stage2==stage3 bit-for-bit match verification.
  4. Sum up the two strategies in one phrase each. What's the shared idea?
    See the answer
    1. Shrink the seed to human-readable size (make it auditable). 2. Double-compile through a different lineage and compare (DDC). What they share: instead of "make all of it yourself," "trust small, and independently confirm each other."

SummaryTrust small, confirm with diversity

To the threat of Bonus 1 (even from clean source, if it's a descendant of a poisoned binary, the poison remains / trust hits bottom at something you didn't make), the modern age responds in two ways. 1. Bootstrappable / Full-Source Bootstrap: shrink the seed you trust down to hex0's 357 bytes, and build up hex → M2 → Mes → TinyCC → gcc → the GNU system from source alone. If the seed is human-readable, the "binary you couldn't inspect" of Bonus 1 can be audited. 2. Diverse Double-Compiling (DDC, Wheeler): double-compile with a trusted compiler of a different lineage, and compare the result bit-for-bit with the suspect binary. Since the poison is in neither the source nor the other lineage, only the contamination disagrees and is exposed.

DDC's foundation is reproducible builds = a descendant of Episode 3's "bit-for-bit match of stage2==stage3." This bonus shakes hands with the main series in three ways ── Episode 3 (bit-for-bit match), Episode 4 (everything from a small core), Episode 9 (the hand-carved seed). Deeply knowing the structure of bootstrapping became, as-is, the power to protect it. The poison Thompson showed to be "undetectable in principle" became detectable, by trusting small and confirming with diversity. ── The chicken-and-egg hope withstands the threat too.

This document is Bonus 2 of "A History of UNIX That Clicks." Historical facts and technology: The Bootstrappable Builds project aims to minimize the number and size of binaries (seeds) that must be trusted. In GNU Guix's Full-Source Bootstrap (2023), from an approximately 357-byte hex0-seed binary, stage0-posix builds hex0 → hex1 → catm → hex2 → M0 → cc_x86 → M1 → M2 → get_machine (mescc-tools) → M2-Planet, then proceeds via GNU Mes (a Scheme interpreter plus the C compiler MesCC), then a self-hosting TinyCC with minor patches, and on up to a conventional C-based bootstrap (gcc and so on). hex0 is a minimal assembler that reads space-separated hex and outputs bytes. Diverse Double-Compiling (DDC) is a technique by David A. Wheeler: compile a compiler's source twice with a trusted compiler of a different lineage, and compare the result bit-for-bit with the binary under examination; if they match, this shows (under certain assumptions) that the binary corresponds to the source. 2005 ACSAC paper (informal), 2009 PhD dissertation (formal proof). As a premise for DDC, reproducible builds (a bit-for-bit identical binary from the same source) are important. The numbers and composition can change with the version of the project. Trust in the hardware/microcode (Episode 10), securing an independent second compiler, and so on remain separate challenges. This article is an explanation for defensive and educational purposes. ── To print, use your browser's "Print" and "Save as PDF" (in the print version, the figures' controls and answers are static/hidden).

Print / Save as PDF: ⌘+P (Ctrl+P on Windows). On screen, use the slider in Figure 1 for the build-up from the 357-byte seed to the GNU system, and the buttons in Figure 2 to try DDC (the comparison fails only when X is poisoned). Click "See the answer" to open each solution.