How it was built

One bug in four costumes2026-08-14

No code tonight. Tonight was for reading two reviews that arrived the same day — one from the principal-engineer seat, one from the research team — and discovering they had converged, independently, on the same sentence: the four consensus bugs that have eaten the last two weeks were not four bugs. They were one.

Each had looked different in the field. A fork-choice that oscillated between committed chains. A maturity handoff where cheap identities head-counted their way into a quorum. Two honest proposers who signed at the same height and slashed each other. A launch gate that let one free anchor bless a competing block. Four multi-region field runs, four attributions, four fixes — and each time, the uneasy feeling of walking in a circle.

The reviews named the circle. Every one of those defects was a finality quorum that failed to intersect over the validator set it claimed to represent. That is the oldest safety theorem in Byzantine consensus: two quorums that share an honest member cannot finalize conflicting blocks, because an honest member never signs twice. Sized against a shifting set, counted by heads instead of weight, signed without writing the ledger, or filled from a larger pool than it was sized over — four doorways into the same room. The feeling of circling wasn't failure; it was what re-deriving a known theory one edge at a time feels like from the inside. The perimeter is finite. We had been discovering it with the most expensive, slowest, least deterministic fuzzer available: a multi-region network under real WAN weather.

So tonight's decisions are about never doing that again. The invariant set — five statements, each annotated with the scar that proved it and the code it governs — is now canon, and every consensus-touching change must say which invariants it touches and why it preserves them. Alongside it, a new first gate: a deterministic model-checker that drives the real consensus core through adversarial schedules — delays, partitions, crash-restarts, equivocators — and asserts all five invariants after every step, on a laptop, in seconds. Its acceptance test is honest: run against the code as it stood before each of the four fixes, it must go red on all four. The proof it would have caught them is the proof it was worth building. Field runs return to the job only they can do — proving liveness on a real, hostile internet — and stop being the place safety theorems are discovered.

The other decision was about restraint. The research certification for the fourth bug corrected our proposed threshold by exactly one: a launch block now needs a strict majority of anchors behind it, because the clever-sounding half-rounded-up version admits a split that is not merely a fork but a permanent partition. One-off-by-one, at the heart of a quorum rule, found by arithmetic instead of by a field run — the whole evening's argument in miniature. Consensus is where this project spends no novelty at all. The invention budget belongs to the part of silt that is genuinely new; the consensus layer's highest ambition is to be indistinguishable from the literature.

It was also a night for admitting what the documentation had become: one viewpoint per bug arc, accreted, with the current plan smeared across consult files and issue threads. The closed arcs move to the archive; the live tree gets rewritten to say one thing. A project should be readable the way its chain is supposed to be — a single head, with history behind it, not four competing tips.

The substrate holds — a night on the P0 floor2026-08-13

The rescue redrew the map: stop re-running the cloud, prove the substrate. This was the night of proving it. Not the flashy consensus corners — the plumbing underneath them, the part that has to work before any red-team can even begin. By morning the floor is solid in every place a laptop can reach, and honest about the one place it can't.

The night's throughline is a discipline the rescue paid dearly to learn: attribute before you build. Four of the five substrate items turned out to already exist. The concentration metric that measures whether a Sybil cohort could quietly capture consensus — built, tested against skew and split fixtures, printed on every commit from the committed ledger, not gossip. The publish path everyone feared was silently stranding content — already durable-or-loud, already confirming that every erasure stripe kept enough shards to rebuild before it hands back a link, already covered by five sim tests. The cold-start re-mesh that a joining node needs when it starts before its seed is listening — already certified over real TCP. Had we taken the audit's file references as diagnoses rather than hypotheses, we would have "fixed" three things that were not broken. Reading the code first is the whole of the method.

What was genuinely missing was smaller and sharper. The dead-holder lifecycle from the day before got its companion: a way to certify the substrate under an adverse network, deterministically, on the loopback — so that "the network stays live and serves under jitter and loss" is a command you run in two minutes, not a hope you hold about a cloud. That harness immediately earned its keep: it caught a bootstrap test trying to pass under netem that had no business there — a clean-localhost timing test with a half-second timeout and no retries, green once by luck, red under loss. The honest move was to exclude it and say why, not to loosen a threshold until it passed. Never fake green, even against your own harness.

The retry-versus-evict rule got a guard that can't flake: a dead peer must be dialed as many times as its retry budget before the network gives up on it. Evicting a good peer on a single dropped packet — the mistake that once starved consensus under loss — now turns a specific test red. And the last publish residual, a publisher that lost its anonymity cover when the one validator it asked had just restarted, learned to ask the next one; the ranking is the same from everyone who holds the chain, so the fix costs nothing and closes the gap.

The honesty this log exists for is the ending. One corner is not green, and no amount of substrate work will make it so tonight: the objective chain, under real multi-region latency, still reorganizes its own committed blocks onto a lighter fork. That is a consensus rule and a published claim, and the discipline is explicit about where those go — to research, not to an overnight guess at fork-choice weight. It is characterized, consulted, and detected by the convergence test itself, so the morning's field run will see it rather than paper over it. The floor is ready. The one load-bearing question left is not ours to answer at two in the morning with a knob.

Finishing the BFT chain2026-08-13

The one corner the substrate night couldn't green was the chain reorging its own committed blocks. Research answered it, and the answer was not a new mechanism — it was finish the one you already have.

The diagnosis was elegant and a little humbling. Silt's objective chain is ninety percent a bond-weighted BFT chain: it gathers a Byzantine quorum, it locks validators to the first block they sign, it calls the result "committed." But bolted on top was a Nakamoto heaviest-chain fork-choice — follow the heaviest fork — and during the first few blocks of a fresh network that rule is fed a weight of zero. The launch anchors, the training wheels a young network leans on, were deliberately given eligibility but no weight, on the principle that only real staked bond should ever move fork-choice. Correct in the mature regime; catastrophic in the bootstrap, because a heaviest-chain protocol whose weight is zero has, for those blocks, no fork-choice at all. It fell through to a tiebreak that compared head hashes — and a genesis block whose hash happened to sort low would win against a two-block committed chain. That is how "committed height two" became "new head height zero." Not a heavier fork winning; a coin-flip promoted to a rule.

The local simulation had passed the whole time, and that too was the two-substrate discipline earning its keep: the sim only ever tested an established chain with real, settled weights, where the tiebreak is the rare symmetry-breaker it's supposed to be. It never modeled the ramp. So the first thing built was not a fix but the missing scenario — a deterministic bootstrap test with zero-bond anchors that reproduces the reorg-to-zero on a laptop, in a millisecond, red before the fix. Only then the fix.

Two of the three defects landed tonight, and they are the ones that restore convergence. Anchors now carry a fixed bootstrap weight — their eligibility is their weight during the young window, which is exactly what the training-wheels principle always meant; it vanishes the moment the network matures, so nothing about the mature security argument changes. And the tiebreak now respects height: a shorter fork can never win a weight tie against a taller one. With those two, a committed chain genuinely outweighs a genesis fork and the oscillation stops. Alongside them, the Byzantine quorum is now sized against a stable validator set — the fixed anchor set during the ramp — instead of a live count that climbed block-by-block as bond registrations drained in and left no fork ever holding a quorum of a consistent set.

The third defect is the interesting one, and it is deliberately not here yet. The owner ratified the bond-weighted BFT reading of the chain, which means a quorum-committed block is final — never reverted, the way Tendermint means it. Enforcing that is a small change, a rolling finality floor generalizing the checkpoint machinery the chain already ships. But it rewrites what objective fork-choice means — from "heal to the heavier fork" to "a committed block is final" — and the red-team tests that encode the old healing behavior are, on inspection, built on validators double-signing, which under finality is a slashing event, not a heal. Changing consensus safety tests because the model changed is legitimate, but it is not something to do quietly at two in the morning while the owner sleeps. So it waits for daylight and a second pair of eyes. The chain converges tonight; it becomes provably final in the morning.

The WAN rabbit-hole, and climbing back out2026-08-12

For two and a half weeks this build log went quiet. That silence is the story.

What happened in the gap was a single bug — a fresh three-region chain that would not commit its first block on a real wide-area network, while the identical block committed in seconds on a laptop. It has a name, #286, and by the end it had a clean one-file fix: a Merkle proof that was quietly O(n) instead of O(log n), rebuilding whole subtrees on every call, so that on a small cloud VM the bond-proof work saturated the CPU and starved consensus of the cycles it needed to gather a quorum. Cache the tree once, proofs go O(log n), a 64-megabyte answer drops from 743 milliseconds to eight, and the chain commits in seventeen seconds. Byte for byte the same proof; the security argument untouched. A good fix.

The trouble was everything before it. That one bug was re-diagnosed across five different theories, and each new theory was discovered by spending a real, billable cloud run. First it looked like a transport deadline cutting the gather short. Then like the validators never learning each other's addresses. Then like an eight-megabyte genesis block too fat to cross the network. Then a size-aware deadline was shipped as "the fix" and a fresh cloud run was spent to prove it wasn't. Only last did the profiler show the real culprit, and by then the meter had run five times over. Worse: at least two of those "layers" were perfectly reproducible on a single machine — the eight-megabyte block was deterministic; a laptop would have shown it for free. We paid the cloud to learn what a local test already knew.

There is a specific discipline this violates, and the sharper irony is that we were writing that discipline down at the same time we were breaking it. Two build-immutables were added mid-thrash: build for the adverse internet (never invent a timeout out of thin air — the mature networks solved this decades ago, so read the settled answer first) and root-cause before you patch (name the failure mechanism in one honest paragraph — it fails because X; this addresses X by Y — before touching a knob, and if you cannot, stop and instrument rather than guess). An expensive run is for confirming a fix you already understand and have reproduced cheaply. It is never for discovering a cause, and never for testing a guess. We knew this well enough to canonize it; we did not yet know it well enough to follow it under pressure.

The tell was the silence itself. A subsystem that eats dozens of commits in a week and produces no build-log entry is not making progress you can narrate — it is stuck in a loop, and the loop processes its own pain by re-running the same failing test rather than by stepping back. Meanwhile the actual forward work — the privacy layer, the takedown layer, the demand receipt, the concentration metric that is the whole point of the next phase — got not a single commit.

So this entry is also the climb out. The compute bug is closed and its regression guard is in place. The remaining wide-area work is folded into one tracked item with one explicit finish line — a single clean warm run, once, as a gate — instead of an open-ended invitation to keep poking a cloud. The adversarial tests that a live cloud kept failing to even set up are being moved onto a deterministic local harness that impairs its own loopback with the same jitter and loss a real network has, so that an attack we want to prove denied is one we can schedule and run on every build, not one we hope fires somewhere over a WAN. And a gate now stands in front of the cloud itself: no billable run without a written mechanism and a named local reproduction first.

The lesson compresses to one line, and it is worth the two and a half weeks only if it sticks: when the build log goes quiet under a pile of cloud commits, the loop is stuck — stop, instrument, reduce it to something a laptop can reproduce, and spend the expensive run only to confirm what you already understand.

Substrate first: starving the dial-storm2026-08-12

After the rescue, the plan changed shape. The instinct all along had been to keep re-running the multi-region cloud test until the trust plane's adversarial drills finally graded green. The principal engineer's reading was sharper: the drills weren't failing on the cryptography. They were failing because the substrate underneath them — the plain business of publishing a file, repairing it, and finding who holds it — was too flaky over a real wide-area network to even schedule an attack. Prove the substrate first, and the rest becomes drivable. So the first forward work is not consensus. It is durability plumbing.

The dominant wound there has a number, #277, and a shape everyone who has run a peer-to-peer network will recognise: the dial-storm. A holder leaves — a laptop closes, a node dies — but the rest of the network keeps a little note saying "that machine has a copy of this chunk." Nobody tears the note up. So every time someone wants the chunk, they dial the machine that is no longer there, wait out a full timeout, and only then try someone else. Under churn a caretaker sweeping thousands of chunks spends all its time dialing corpses and never gets far enough to notice the real loss, let alone repair it.

The honest part of this fix was the part before any code. The audit's best guess was that the discovery walk re-dialed dead holders. But reading the code showed that walk had already been fixed weeks earlier — it skips a peer it recently failed to reach. And signed provider records already carry an expiry that the fetch path already honours. So the hypothesis was half-stale, and building against it would have fixed nothing. What was actually still broken was subtler and worse: the negative cache that suppresses a dead holder only holds for thirty seconds. After that the holder is dialed again — because its note in the provider store is never removed. The cache wasn't stopping the storm; it was rate-limiting it, to one wasted timeout per holder every thirty seconds, forever. And two other paths — the one that hands provider records to other nodes, and the one that re-announces them — never checked the cache at all, so the network kept teaching itself about holders that were already gone.

The fix gives a provider record a life, and an end to it. A departed holder's record now ages out on a lease it must keep renewing by re-announcing; a node stops serving records it knows are stale; and the moment a holder is confirmed dead — not merely slow, but past all its retries — its record is pruned from every chunk that still has another live holder. That last clause is the whole art of it. You cannot simply delete a dead holder everywhere, because sometimes it is the only holder a chunk has, and orphaning that record would make the content undiscoverable the instant the machine blinks — which is exactly the restart-survival bug this project has fought before. So a sole holder's record is kept and kept being re-probed, patiently, in case it comes back; only where a live sibling already exists is the corpse dropped. Replicated content, which is the norm, sheds its dead weight; the fragile single-copy case keeps its lifeline.

And because the next milestone after "does it work" is "does it work efficiently," the plan says to start counting now, before any of it is tuned. So two gauges went in with the fix: how many dead-holder dials the cache avoided, and how many stale records were aged out. They are the early-warning lights for the day some future change quietly reopens the leak — the number climbs, and you know where to look. You cannot make efficient later what you never measured now.

One small scar worth recording, in the spirit of this log: mid-change, a stray command to undo a throwaway edit reverted a whole file of real work along with it. The failing-first test had already done its job — it went red exactly where it should when the fix was absent — so the loss was visible immediately and cost minutes, not hours. That is the entire case for writing the test first, made in miniature.

Meeting across networks without becoming the network2026-07-26

Everything worked on 127.0.0.1. Two nodes in the same house found each other and talked. But two nodes on separate home networks could do neither of the two things they need to: they couldn't find each other (rendezvous), and they couldn't connect (NAT — a home router lets you dial out but never accepts an unsolicited dial in).

The first design decision was to split those two problems apart, because they have different answers and conflating them produces mush. Rendezvous is "how do I learn a peer exists and where to aim." NAT traversal is "the socket won't open even though I know where to aim." We built them as separate rungs: mDNS for the same-house case (free, no infrastructure), a reachability check (our AutoNAT — a node asks a couple of peers to dial it back, and believes it's publicly reachable only if one actually lands), and then, for the hard case where both peers are NATed, a relay.

The second decision was the one with teeth, and it's a values decision as much as a technical one. A relay is the natural place for a project to plant infrastructure — "connect through our relay." We refused that. In Silt a relay is a node capability, not special infrastructure: any publicly reachable node can offer -relay, none is privileged, and no relay address is baked into the binary. The public dev box we test against lives in a separate deploy/ directory, clearly labeled as throwaway scaffolding the project does not operate. The network must not depend on us to exist.

What makes that stance safe is that the relay is content-blind by construction, not by promise. A relay splices two raw byte pipes; the sender then runs its normal pinned end-to-end TLS handshake with the target through the splice. So Silt's core security invariant — "a frame's sender is whoever the TLS handshake authenticated" — holds unchanged across a relay, because the relay only ever moves opaque bytes it cannot read, alter, or forge. It learns metadata (which two node IDs talked, when, how much), the same an on-path router already sees, and the threat model says so plainly.

Then reality taught the lesson it always teaches. The first real cross-network run — one Mac, one relay box, one field test — failed, and the failure was invisible in every in-process simulation we'd ever run: the transport dialed a fresh connection per message, so a reply to a NATed peer required dialing into it, which is exactly the thing NAT forbids. Bootstrap came back with zero peers and no error. The fix was to make replies ride the connection the peer itself opened, and to stop stamping undialable wildcard addresses into peers' address books. It was a transport bug, in transport code — precisely the class the sim can't see because the sim never opens a socket. That single field test is why there is now a multi-process end-to-end suite that runs real daemons over real TCP in CI: so the next bug of that shape dies in a pull request instead of on a kitchen table across town.

The placement spectrum, and why we moved off one end of it2026-07-25

For a long time Silt placed every chunk independently, by its own hash. That's the maximum-fanout extreme of a spectrum, and it has real virtues: perfect spread, dead-simple logic, no coordination. But it has two costs that only show up at scale, and both are the kind of thing you must fix before a live network of independent operators ossifies the wire format — migrating them through a placement change later is far more expensive than doing it while the network is empty.

Cost one: reads got expensive. A file of S stripes cost roughly S × k separate lookups-and-fetches, because every shard was found on its own. Cost two: concentration risk wasn't actively managed. Nothing stopped two shards of the same stripe from drifting onto the same node, or the same failure domain, as churn reshaped the network — and if that happens, one machine's death can cost you two shards of a stripe instead of one.

The fix was to stop thinking in chunks and start thinking in columns. A column is one shard position across all stripes of a file. Place, find, repair, and audit by column — keyed by hash(root‖j) — and three things fall out at once:

• A column's providers are found in one lookup, not S of them.
• Losing one unit costs exactly one shard per stripe — you can lose up to n−k whole columns and still rebuild every stripe.
• Within-column anti-affinity becomes structural, not a heuristic you hope holds.

Then we pushed further, because "don't put two shards on one node" isn't enough — you want "don't put two shards in one failure domain" (an autonomous system, a rack, a datacenter, an operator). Nodes now carry a -domain label and gossip its hash on every message, the same way they gossip their capacity pledge. Publish steers each column onto a domain no other column has used; repair re-seeds rebuilt columns into domains the survivors aren't using. It's best-effort and never a veto — a node spreads across the domains it has actually learned about — but on an identical layout it roughly halves worst-domain co-residence versus domain-blind placement.

The through-line: the earliest, simplest thing (place-by-hash) wasn't wrong, it was one end of a spectrum. Naming the spectrum let us choose a point on it deliberately, and — crucially — choose it while the cost of choosing was still just a code change, not a network migration.

One process, but a real product from day one2026-07-23

The temptation, building a distributed storage network, is to reach for a fleet of servers immediately — spin up VMs, wire real sockets, watch nodes gossip. We did the opposite, and it's the single decision the whole codebase still rests on: the entire network runs in one process, and will keep doing so until it genuinely needs real sockets.

The catch that makes this honest rather than a shortcut: the code is structured so that swapping the in-process simulation for real TCP/QUIC/libp2p transport later touches only adapter code, zero core logic. That's hexagonal architecture — ports and adapters — taken as a hard rule, not an aspiration:

• The core domain — chunking, crypto, erasure coding, manifests, DHT routing, the repair policy — is pure logic. It imports no networking, no filesystem, no wall clock, no global RNG. Every effect leaves through an interface defined in a shared ports package.
• Adapters implement those interfaces: in-memory ones for the sim, real ones (tcpnet, diskstore) for production. The core can't tell which it's talking to.
• The simulator isn't a mode threaded through the code — it's just a harness that wires nodes together with in-process adapters and drives a simulated clock.

Two rules keep it from rotting. First, a dependency-lint test fails CI if a core package ever imports an adapter — the boundary is enforced, not trusted. Second, determinism: every source of nondeterminism (time, randomness, message ordering and latency) is injected, so a sim run with a given seed produces byte-identical results. That last property is worth more than it sounds. When a churn bug shows up 40 nodes and 10,000 messages deep, "re-run with seed 0x5117 and watch it happen again" is the difference between a fix and a shrug.

The payoff came later, and it was pointed: when we finally did leave the sim for real machines on real home networks, the swap touched adapters and left the core alone — exactly as promised. The bugs we found there (see the cross-network entry) were transport bugs, in transport code, where they belonged.