ADR-0020: Sequencer hand-off

ADR-0004 makes the sequencer's identity a key, and ADR-0012 makes two receipts from that key for one (batch, position) with different contents a slashable…

Status: accepted (accepted in part). Dated 2026-09-14.

Status: accepted in part, 2026-09-14. The first half is built; the second is the owner's decision and is stated with its cost.

The hazard

ADR-0004 makes the sequencer's identity a key, and ADR-0012 makes two receipts from that key for one (batch, position) with different contents a slashable equivocation — demonstrated with a real slash. That is the right rule against a dishonest sequencer, and it does not care why the two receipts exist. Two honest nodes running under one key produce exactly the same evidence:

  • A copy left behind. A standby is provisioned from a datadir copy; the primary keeps sealing. If the standby is ever started from that copy, its counter says "next batch N" while L1 has moved on to N+k, and its first block re-issues batch N's numbers under the same key. Every receipt it hands out is a slashing proof against itself, paired with a receipt the primary already gave someone else.
  • Two at once. Primary and standby both up, both at N. Each seals batch N locally and signs its receipts; the first to reach L1 wins the open_batch, the second's settlement fails — but the receipts were signed before either reached L1, so the equivocation exists whichever one loses.

DEPLOYMENT.md §9 item 8 asked for "a standby that can take over the receipt identity without equivocating (signed hand-offs)". A signed hand-off record does not by itself prevent either case: a record says what the primary did last, and the hazard is what the standby does next on stale or concurrent knowledge.

Built: the copy left behind cannot sign

Two changes in the node, 2026-09-14:

  1. Before signing anything, the node compares its own next batch with L1's. stale_copy_check(local_next, feed_next_seq): the DA feed's next_seq is the only batch number L1 will open next, so it is also how far any node has carried this identity. Level or ahead is a node resuming its own work — the unsettled blocks re-queue as before. Behind is a copy left behind, refused by name with the cure: bring the datadir up to date, never run two nodes under one key. No override flag, because there is no honest reason to sign from behind.
  2. A datadir without its identity file resumes its batch counter from the log, not from zero. Batch k carries block k+1, so the next batch is the number of blocks logged. Before this, a datadir rebuilt from the published bytes — the runbook's own recovery path — would have opened batch 0 again under a key that had already sealed it, which is the first hazard reached by a different road.

What this closes: the stale-standby case, in the node, before any receipt is signed. What it does not close: two nodes at the same height at the same time.

Proposed: claim the batch on L1 before the first receipt in it

The structural fix for "two at once" is to make the batch number a claim on L1 before it is signed for, not after. The DA program already has the claim: open_batch(k) succeeds once. Today the node signs receipts for batch k as transactions arrive and opens k on L1 only when it settles, so there is a window — the whole life of an open batch — in which a second node can sign the same numbers. If the node instead sent open_batch(k) when it opened k locally, and accepted no transaction into k until that landed, a second node's open_batch(k) would fail and it would accept nothing: no receipt for k could exist from two nodes, because only one of them could ever open k.

What it costs. One more L1 transaction per block in the combined mode ADR-0017 introduced to save exactly that: the combined settle packs open, chunk and seal into one transaction, and an early open splits it into open now and chunk-plus-seal later — about 5,000 lamports per block at the base fee, against the ~4 M lamports a block costs today with rent still sunk, and against the 5,000 flat fee a one-transaction block earns. It also puts one L1 round trip in front of the first admission of every block, which is latency the receipt's soft confirmation currently does not pay.

What it buys. The one property a standby needs and no record can give it: two honest nodes under one key cannot both sign. With it, a hand-off is an operational procedure with no protocol content — stop the primary, start the standby, and the worst a mistake can do is fail to open a batch.

What it changes for the receipts. Nothing in their bytes. A receipt still names (chain, batch, position, slot); the batch it names is now one the signer holds on L1 at the moment of signing, which is a stronger statement than today's, not a different one.

This is the owner's call because it moves cost and latency, and it belongs with ADR-0017 §6.2's block-interval decision: a longer block interval amortises the extra transaction the same way it amortises everything else.

Until then: the operating rule

One key, one running node, always. Provision a standby from the datadir and without the identity file (sequencer.json) — a standby that cannot sign cannot equivocate. To fail over: stop the primary, confirm it is stopped, copy the identity file over, start the standby; the stale guard above refuses the start if the copy fell behind, and the log resume above makes a rebuilt datadir start at the right batch. What the rule does not survive is a primary that is not actually stopped, which is what the claim on L1 is for.

Not decided here

Whether the claim, once built, should be the DA program's open_batch or a separate lease PDA with an expiry, so a dead primary's open batch does not block a standby until someone abandons it. abandon_batch exists, but only the feed's poster may send it (protocol/onchain/programs/solieum-da-program/src/lib.rs:190-195, NotThePoster), so it frees the number for any holder of the poster key; whether that is enough is a question for the drill that DEPLOYMENT.md lists and nobody has run.

Corrected 2026-10-09: the sentence above called abandon_batch permissionless. It has required the poster's signature since the program's first commit (6035443); the amendment below says so too (14cb981). Drill 09 (e0d3071, 2026-10-09) has since run the fence and the hand-off on throwaway chains; it did not exercise a claim on L1 or a dead primary's open batch.

Amendment, 2026-10-09: the fence and the hand-off

Status of this amendment: built, in no fleet build. No program, no account layout, no receipt format and no commitment version changes. Working notes: protocol/docs/notes/sequencer-failover-2026-10-09.md.

Two corrections to the text above

  • The claim proposed above cannot be sent as written. open_batch takes the batch's frame_hash and total_chunks, and neither exists before the batch is sealed. An early open must carry a placeholder and be abandoned and re-opened at settle. And abandon_batch is not permissionless: it is the poster's, and it frees the number for any holder of the poster key, which a primary and its standby both are. A claim made of today's instructions is a lease only between nodes that both honour it.
  • The node had a third way to re-issue a batch. Block production writes the block log, then the identity file. A stop between the two, or a replica that copies the two files at different moments, left nextBatch a batch behind its own log, and open trusted the file: the next admission signed the last logged batch's slots again. The counter now never resumes below the highest batch the log's receipts name, and a file counting more batches than the log has blocks is refused at open.

Built: a fence, and a hand-off that waits for it

  1. The fence. fence.json in a datadir. A node reads it before every receipt and on every producer tick; once it is there the node signs nothing, drains nothing from L1, seals what it had already signed, and acknowledges in the same file with its next batch and head block. A fenced datadir does not start as sequencer (exit 3). The DA feed is a second source: a feed whose next_seq passes the node's batch shows a batch sealed elsewhere under this identity, and the node fences itself. Only a hand-off to a directory clears its fence.
  2. The hand-off. solieum-node handoff --from PRIMARY --to STANDBY fences the primary and waits for its acknowledgment, or, for a primary that crashed and cannot give one, for its process to be gone (--primary-pid) or the operator's word (--primary-stopped), never while its RPC still answers. It then requires the standby's block log to be the primary's block for block and its journal to be the primary's, refusing a standby behind by name (or, with --sync, copying what it lacks) and refusing for good one that sealed a block of its own. The identity file is copied last, a promotion record is written, and -- run ... starts the standby in the same process. A promoted directory that later resumes below its record refuses to open.
  3. A crash mid-batch loses nothing: the receipts the primary gave out are in its journal (ADR-0024), the hand-off carries the primary's journal, and the standby restores it verbatim and continues at the next position.
  4. solieum-node receipts-audit reads every receipt in a set of datadirs and judges any slot filled twice with the receipt crate's own prove_double_assign, the judgement the bond program slashes on.

The in-process tests build every schedule of submit and seal around a hand-off that three steps allow (the replica taken at any point, the fence landing at any later one, the primary acknowledging or crashing, 160 schedules) and find no slot filled twice and no batch skipped; the control that copies the identity without the hand-off does fill one twice, and the audit names it.

Still open

What the hand-off refuses, it does not make safe: a primary that neither reads its datadir nor sees L1, and cannot be shown to have stopped, keeps the identity, and the standby waits. Promoting without reaching the primary needs a claim on L1 before a batch's first receipt, either a lease built from today's instructions (one more transaction per block, honoured by convention) or one the DA program enforces (a program and feed-layout change). Both remain the owner's decision, with the cost argument above. Also open: the settle path takes an L1 batch or root under its block number as its own without comparing contents (the "already sealed" and "already committed" skips), which matters only to a node that sealed a different block under that number; the hand-off's checks keep a standby from doing so.

2026-10-11: both guards fail closed on a feed they cannot read (K52)

Found from source by drill 09's agent and not exercised: the stale-copy check and the L1 fence read the DA feed through the backend's read, which turns a refused quorum read (the readers disagree, or none answers) into "absent", and both took "absent" for "no batch sealed elsewhere". So a stale copy started, and a running node signed on, whenever its readers disagreed about the feed, which a lagging reader does exactly when the feed has just moved. Fixed in the node (4a1aba8; not gated, in no fleet build):

  • run takes up to three readings 5 s apart and then refuses to start (exit 3), naming the cause and the cure, with no override.
  • A running node pauses on the first refused reading: it signs no receipt, drains nothing from L1, writes nothing to its datadir, and raises getSettlement.alarms.fenceBlind. The first reading the readers agree on lifts the pause, or fences the node if it shows the feed past its batch. The pause is in memory only; it is not a fence and nothing acknowledges it.

What it costs: an L1 read outage that refuses the feed now stops admission for as long as it lasts (settlement already stopped for it), and a restart during one is refused. Why one refusal and not more, and why no backoff beyond the 5 s cadence, is in failover::FENCE_BLIND_READINGS. It leaves the amendment's open items as they were: the fence still sees a second signer only once that signer's batch is on L1, so receipts both signed before then remain a slashable pair, which only a claim on L1 before a batch's first receipt prevents. An embedded L1 is read in-process, never refuses, and is unchanged.