Arca — HA Hardening (remediation after the Phase 29 review)¶

Progress Overview¶

R1

R2

R3

R4

R5

R6

R7

R8

R9

Context¶

The Phase 29 review (arca-phase-29-ha-review.md, passes of 2026-06-10 and 2026-06-11) produced the findings: 3 P0s (§2.1–§2.3), §2.4 + 9 P1s (§3.x, D1, D2, D3a, D9), a security-critical chain (§3.7), a series of P2s (M1–M8, §5, D3b/c, D4–D8) and P3/doc (D10–D12). This plan implements ALL of them, organized in 9 milestones (R1–R9) ordered by priority and technical dependency. It is a living document, published on the documentation site as an annex of the roadmap (Phase 29.1) via a symlink to the canonical file .claude/plans/arca-phase-29-ha-hardening.md.

Security workstream (review §3.7). The single most important security gap — a rogue peer receiving all new data with no secret, because the fan-out push direction authenticates no peer — plus its amplifiers (brute-forceable fingerprint, weak secrets allowed, plain-HTTP/unverified-TLS/no-replay) is spread across R3 (peer authentication of liveness + fan-out gating; fingerprint off the public endpoint) and R4 (verified inter-node TLS / TD-015; secret strength; anti-replay). On a deployment whose cluster network is not a trusted, isolated segment, treat the peer-authentication half (decision H12) with P0 urgency and consider pulling R4's TLS item forward to sit beside R3. The secret itself is well handled where it is used (never on the wire, HMAC-keyed SigV4, isolated credential); the gap is that authenticating peers — not just requests — was never designed.

How to use this document (process rules, valid for every session):

At session start read: CLAUDE.md, this plan, the review. The traceability table at the bottom says what is done and what is not.
Work TDD; every milestone must leave bin/test unit and bin/test cluster green. After code changes also rebuild the test images (docker compose -f docker/docker-compose.yml build unit-test test), otherwise tests run on stale images.
For every completed item: tick the checkbox HERE and in the traceability table, update CHANGELOG.md (Unreleased section), update the documentation touched.
If a decision changes the design, update the WHOLE document, not just the touched section (same rule as the Phase 29 plan).
At the end of each milestone: turn the milestone's cell green in the progress bar at the top of THIS document, tick the milestone in the roadmap's Phase 29.1 section (documentation/docs/roadmap.md), rebuild the site with bin/docs-build (this plan is published there via symlink), then propose to Pietro a commit + possibly a release.
The §x.y / Mx / Dx numbers refer to the review; the "plan lines" to .claude/plans/arca-phase-29-ha.md.

Design decisions (fixed before implementation)¶

#	Decision	Status
H1	True quorum = option A of review §2.1: count the fan-out ACKs; if `1 + ack < write_quorum` the write fails with `503 ServiceUnavailable` + `Retry-After`. The local copy is NOT rolled back (anti-entropy propagates it): document that an error does not imply undo, as in every quorum system without distributed transactions.	✅ approved (review + Pietro's ok on the P0s)
H2	ACK = row applied + blob present: the response of `POST /cluster/v1/object` becomes `{applied, has_blob}`; the peer self-certifies (sidecar present for rows referencing a blob; for delete markers/tombstones `applied` suffices). Avoids threading state between `write_sidecar` and `put_object`. For composites `has_blob` = composite sidecar present (the parts already fan out individually; the rest is covered by D4).	✅ decided in planning
H3	PG commit-ordered cursor via counter table (like SQLite): table `object_seq(v BIGINT)` updated with `UPDATE ... RETURNING` in the same transaction as the row. The row lock serializes the assignment until commit → seq order = commit order, no deliverable gaps. Cost: serializes the final stretch of object writes on PG (acceptable; documented). Discarded alternative: a probabilistic guard window.	✅ decided in planning
H4	Scope of the ACK counting: object data-plane mutations (put_object, version-delete/tombstone, delete marker). Control-plane and tag ops remain best-effort fan-out + reconcile (rare mutations, reconciled; extension possible later).	✅ decided in planning
H5	Symmetric leader gate for the workers: only the node with the lowest `node_id` among the eligible nodes (membership + self) runs the gated worker. Automatic failover at the next tick. A double-execution window during membership disagreement: accepted and documented (idempotent work). Refined in R6 (2026-06-12, both points confirmed by Pietro): (a) the predicate is `eligible()` — not merely alive — for the same reason the quorum gates on it (an unauthenticated rogue with a low `node_id` must not steal the role and silence the workers cluster-wide); (b) the gate applies to the lifecycle worker only — the original idea of also gating the Phase 28 replication worker rested on a wrong premise in review §3.3 (see the R6 section).	✅ decided in planning; refined in R6
H6	D3a = gate, not just a warning: in quorum mode, if the observed live nodes exceed `cluster_size`, writes are refused (503 with a clear message) + flag in `/admin/cluster`. It is a misconfiguration that enables split-brain: the safety mode must fail closed. No config escape hatch (rejected as a foot-gun if left enabled); the R9 runbook documents the correct resize procedure instead.	✅ confirmed by Pietro (2026-06-11)
H7	Drift = out of the quorum: a peer with `config_ok=false` does not count toward the quorum (it stays visible in `/admin/cluster` with the drift flag).	✅ decided in planning
H8	Secret rotation via dual-secret: optional `[cluster] secret_previous` key; inbound auth accepts both, outbound uses only `secret`. Runbook: set `secret_previous=old, secret=new`, rolling restart, then remove `secret_previous`.	✅ decided in planning
H9	D6 console via server-side proxy: a `?node=<node_id>` parameter on the per-node admin endpoints, internal proxying via `ClusterClient`. Avoids CORS and browser-unreachable endpoints (typical deployment: only the LB is exposed). As confirmed, three refinements: (a) transport = server-side proxy (direct browser-to-node calls rejected); (b) scope = ALL FOUR node-local view families — audit log, metrics history, notification event log, AND the Phase 28 replication journal (node-local by design, same behind-the-LB problem, added beyond the original plan); (c) the node selector also offers an "All nodes" merged view (entries from every live node fused by timestamp, each labeled with its source node), not just single-node selection.	✅ confirmed by Pietro (2026-06-12): proxy + all 4 views + "All nodes" merge
H10	Rolling upgrade across mixed versions: every wire change is additive (new JSON fields ignored by old nodes); the probe uses the new authenticated `/cluster/v1/ping` with a fallback to the old `/cluster/v1/health` on 404 (legacy peer, counts as alive with a warning).	✅ decided in planning
H11	Versions: a SINGLE release at the end of the full remediation plan (R1–R9); version number decided then (at least MINOR: the quorum changes observable behavior). Supersedes the original idea of `v0.26.0` after R1+R2 with per-milestone releases. Milestones are still committed and pushed as they complete.	✅ revised and decided by Pietro (2026-06-11); version fixed at v0.26.0 (Pietro, 2026-06-12)
H12	Authenticate the peer, not just the request (closes review §3.7). Today the cluster authenticates the sender of every `/cluster/v1/` request (inbound, via `cluster_auth`) but never the receiver* of a fan-out: `live_peers()` filters on `alive` only, and membership admits an mDNS peer on a `cluster_id` match alone — so a rogue peer receives all new writes with no secret. Fix, in order of preference: (1) mutual TLS with a shared cluster CA — a node verifies the peer's CA-signed cert before adding it to membership / fanning out (also resolves TD-015 and the cleartext/MITM exposure). (2) Secret-only challenge-response: an app-layer exchange where the peer proves possession of the secret over a fresh nonce before being counted live — NOT merely "a signed `/ping`" (a rogue controls its own server and can return 200 unconditionally; the peer must prove possession to us). Either way, `live_peers()` (fan-out) AND the quorum count only authenticated and `config_ok` peers. CA origin (confirmed with Pietro): the CA is operator-distributed via config (`[cluster]` CA path + per-node cert/key) with a shipped generator command reusing the rcgen `tls-init` generator — NOT a K8s-style auto-enrollment (first node generates the CA, peers get CSRs signed over a secret-authenticated channel): issuance gated by the standing secret would collapse mTLS to secret strength, while an operator-distributed CA is an independent second factor (K8s mitigates with short-TTL join tokens; possible future evolution, not now). Layering (confirmed): (2) is not a fallback — it ships for ALL clusters as the baseline peer-auth layer in R3 (it also closes the rogue peer on plain-HTTP deployments, where no certificates exist; on a plain-HTTP network it stops the easy attack — a rogue mDNS registrant — while sniffing/MITM still requires TLS), with (1) on top as the independent second factor on TLS clusters.	✅ confirmed by Pietro (2026-06-11): BOTH layers — challenge-response baseline for every cluster (R3) + mutual TLS with the CA via config + shipped generator tooling (R4)

R1 — P0 correctness (true quorum, PG cursor, tombstone-first)¶

The review's three P0s plus the §2.4 prerequisite. No new test infrastructure required (unit + the existing cluster suite); partition-based verification arrives in R2.

Outcome: quorum mode delivers what it promises at ACK time; the PG incremental sync loses no rows; the control merge no longer has the resurrection window.

R2 — Cluster test infrastructure (real partitions, available mode, catch-up)¶

The proving ground of the R1 fixes and of everything else. Extends bin/cluster, the dedicated compose file and tests/integration/test_cluster.py.

Partition helpers in bin/cluster: partition <n> / heal <n> via docker network disconnect/connect (both processes stay ALIVE: that is the difference from node-stop). Required a dual-network compose design: the cluster network carries inter-node + LB traffic and the seed list now uses arca-N-cluster aliases that exist ONLY there, so disconnecting a node severs peer traffic; the never-partitioned mgmt network keeps the test runner connected to every node by its plain service name (without it, the runner would lose the isolated node along with the peers). heal re-attaches WITH the alias (a manual network connect does not restore compose aliases); the node may return with a new IP — peers re-resolve per probe, but HAProxy (startup-resolved) may not track it, so the partition phases are self-contained (fresh up/down -v) and post-heal assertions target nodes directly. Test configs also gained request_timeout_seconds = 5 (not in the drift fingerprint — verified) so fan-out toward an unreachable peer fails fast.
Quorum window test (verifies §2.1) — phase G: partition arca-3 → it refuses writes with 503 with no wait after the disconnect, on purpose: within the window where its membership still sees the peers alive, the missing fan-out ACKs produce the 503 (observed: the failing PUT took ~5 s = the fan-out timeout, i.e. the in-window path); reads still served from the isolated node; the majority side keeps writing; heal → the majority-side write converges on arca-3 and a polled PUT proves it accepts writes again (its own membership view lags a few ticks).
Available-mode overlay (D8, §5.1) — docker/cluster/config-available.toml (mode=available, everything else identical) + docker-compose.cluster.available.yml mounting it on all 3 nodes; phase H: (a) with 2/3 nodes stopped the survivor still accepts writes (same topology where quorum mode 503s in phase C); (b) partition, writes to the SAME key from both sides — both accepted — heal → a single LWW winner everywhere, the loser disappears with no errors (the basis of the §5 doc sentence, to be written in R9).
Control-plane catch-up test (§5.2) — folded into the existing lifecycle: phase B (node 3 down) creates cluster-catchup-cp bucket + a credential via the LB (new signed _admin_post helper); phase D polls node 3 directly until both appear via the control-plane reconcile.
Flakiness fixes (§5.3): wait_live post-convergence sleep 2 s → 5 s (HAProxy fall 2 inter 2s ≈ 4 s to evict); phase E precondition now requires live_node_count == 3 on /admin/cluster alongside the cluster-min free-space check (arca-3's disk stats are cleared while it is considered dead).
7 new pytest markers registered in conftest.py (cluster_partition_before/minority/healed, cluster_available_full/split/converged/minority); "Integration — HA Cluster" row in README.md updated 11 → 24 (+ totals). Full suite green: 8 phases (A–H), 24/24.

Outcome: every consistency claim is exercised by a reproducible test, partitions included. (Release deferred to plan completion — H11 as revised.)

R3 — Membership and quorum integrity (ghost quorum, guards)¶

Outcome: the quorum only counts nodes that can really receive replicas; no more ghost quorum; tombstone GC aware of liveness.

R4 — Inter-node transport security¶

Outcome: an inter-node surface with authenticated peers (verified TLS), a replay window, body limits, strong-secret enforcement and secret rotation without downtime — closing the §3.7 security chain together with R3. Note: the cluster integration suite runs plain-HTTP, so the mTLS chain is covered by the in-process e2e handshake test (and unit tests); a TLS-cluster compose overlay remains a possible §5.4 extension.

R5 — Reconcile completeness (TD-016, multipart, SSE-C)¶

Outcome: all managed state converges after an absence, multipart included; TD-016 resolved; SSE-C read-repair shipped (no TD-017). Bonus finding fixed: the M7 churn was a perpetual full-table redelivery loop between caught-up nodes.

R6 — Cluster-aware workers (leader gate)¶

§3.3 / H5: ClusterState::is_worker_leader() (arca-core — the codebase has no ClusterContext; ClusterState is that object) = own node_id is the lowest among the eligible nodes (including self); always true on single-node. Eligible — not merely alive, refining the original H5 wording — for the same reason the quorum gates on it: an unauthenticated rogue (or a drifted peer) with a low node_id must not steal the role and silence the workers cluster-wide. Exposed as worker_leader on /admin/cluster and /admin/health?verbose=1 (in a stable cluster exactly one node claims it; the R6 tests and the runner's leader discovery consume it, the console can in R8).
Lifecycle worker (worker.rs): in a cluster, only the leader runs the tick (debug-level log when skipping). The deletes it produces stay replicated/tombstoned as today. New optional [lifecycle] interval_seconds TOML knob (default 3600, the previous hardcoded value; validated ≥ 1) — the test configs need a 5 s evaluation cadence, and the interval was not configurable at all.
Phase 28 replication worker — NOT gated (decision confirmed by Pietro 2026-06-12, deviating from the original plan): the review's §3.3 premise ("every node has its own journal and tries to replicate the same objects → duplicate deliveries") does not match the code: the journal is strictly node-local — entries are inserted only by maybe_emit in the S3 handlers of the node that served the client write; the cluster receive paths (/cluster/v1/*, anti-entropy apply) never emit, and the journal is not among the replicated families. Each S3 write is therefore journaled exactly once cluster-wide and duplicate deliveries cannot occur, with no leader needed; gating the worker would instead ORPHAN the entries journaled on non-leader nodes. Decision recorded in the worker's module doc; real RPO caveat documented in ha.md: a node lost for good takes its pending journal entries with it (a plain restart loses nothing — the journal is on disk).
Audit of the other workers (recorded in the worker.rs module doc): retention purge, metrics snapshot, notification delivery = NOT gated (they operate on node-local tables/events by design; the notification queue is fed only by the S3 handlers of the serving node, so deliveries are already exactly-once); the lifecycle's abort-incomplete-multipart = gated with the lifecycle (same tick).
Tests: arca-core unit tests (single-node, lowest-wins, failover-to-next-lowest, ineligible-lower-peer ignored, snapshot flag) + config validation; new cluster phase I (bin/test cluster, +4 tests → 41): with a short-interval lifecycle rule (past-Date expiration — Days: 0 is rejected) the expiry happens exactly once cluster-wide (sum of Lifecycle::ExpireObject audit entries across all nodes == 1; the audit log is node-local, peers apply the replicated delete without auditing) and exactly one node claims worker_leader (the lowest node_id); then the runner discovers the leader via verbose health, stops THAT container (node ids are random — the leader's index differs per run), and the failover sub-phase verifies a single new leader among the survivors and a second expiry, again exactly-once. The runner also gained dump_logs_on_failure: any failed cluster phase preserves the node logs to /tmp BEFORE down -v destroys them (the R5-known M1-class flake recurred during the R6 run and the evidence was lost again; this was pencilled for R7 and pulled forward).

Outcome: no work duplicated N times, with automatic failover of the role; external replication deliveries shown to be exactly-once by construction (per-node journal), not made so by a gate.

R7 — Synchronization state and operability¶

D2 "Syncing" readiness: ClusterState (the codebase's ClusterContext) tracks, per peer, the completion of the FIRST full reconcile pass since startup (objects drained to the manifest end + control snapshot merged, recorded by the anti-entropy worker as PeerSyncStatus.first_pass_done); until it is complete toward every eligible peer (not merely live — a rogue/drifted peer is never reconciled from, so requiring a pass toward it would deadlock readiness), the plain /admin/health answers 503 {"status":"syncing"} + Retry-After: 5 (drain takes precedence; no eligible peers → degraded ok, documented in ha.md; ?verbose=1 stays 200 with status: "syncing"). The LB and the k8s probes see it → a realigning node receives no traffic until it is coherent. As built, the per-peer HWM moved from the worker's local map into ClusterState so the endpoints can expose it; sync entries are pruned with membership (M3) and survive a peer's death (a returning peer resumes incrementally). Documented consequence: a brand-new peer joining flips the established nodes to syncing for up to one anti-entropy tick. Integration: new phase J (bin/test cluster) — a node restarted after downtime holds 503-syncing (observed by polling started right after start, no wait_live) and the catch-up object must be readable the INSTANT the health turns 200 (readiness implies correctness, no polling allowed); wait_live also gained an LB-routes-200 guard (on a cold cluster start ALL nodes briefly sync → HAProxy could have zero backends when tests begin).
Exposed lag: /admin/cluster with, per peer, sync: {hwm, lag, last_reconcile, first_pass_done, skipped_entries} (lag = peer's ping-reported max_seq − HWM) + the node-level syncing flag (also in the verbose-health cluster snapshot).
D3c Rewind detection: the ping's max_seq now travels ContactInfo → PeerNode.max_seq (None for legacy/dead peers); per-tick check via the pure sync_rewound() (arca-core) → HWM reset to 0 + warning. As built, with a freshness guard beyond the plan: probe and reconcile run on independent cadences, so the report must be NEWER than the last HWM advance (peer.last_seen > hwm_at) — otherwise a peer under sustained writes false-alarms with a stale ping taken before the rows we just pulled (a genuine restore keeps reporting the rewound counter, so detection is only deferred to the first post-reconcile probe, never lost).
M1 Stuck HWM: after K=5 consecutive passes failing on the same seq (STUCK_SKIP_AFTER, pure StuckTracker unit-tested per peer/reset/progress), the entry is skipped with a warning + skipped_entries per peer in /admin/cluster. apply_entries/reconcile_peer_objects now return the failing seq alongside the cursor.
M2 Repair budget: repair_blobs with a per-tick budget (new [cluster] blob_repair_budget, default 100, validated ≥ 1) + a resume cursor (sorted blob-id order; None = sweep complete). Budget counts peer FETCHES (the local existence stats are cheap); checked only BETWEEN blob ids so one composite is always processed to completion (bounded overshoot — a composite with more unfetchable parts than the whole budget cannot stall the sweep's progress). A mid-flight sweep continues on EVERY tick (only complete sweeps wait for the slow 10-tick cadence), so a backlog drains at budget/interval. Unit test with a fake blob peer: budget 2 over 4 missing blobs → exactly 2 fetched + cursor, resume completes.
M4 Retry-After: verified and pinned by unit tests in error_response.rs (every ServiceUnavailable carries Retry-After: 5; non-503s carry none — covers quorum, size-gate and any future cluster 503 in one place); the new syncing health 503 sets it explicitly.
M8: tracing::warn! on the mtime→now fallback in fs/blob.rs (walk_files), naming the file and the consequence (a filesystem without mtimes never reclaims blobs).
N2 — equal-timestamp LWW clobber erases Object-Lock state cluster-wide (found during R7; THE root cause of the known phase-D retention flake, which was previously mislabeled M1-class). Lock changes (N1) keep last_modified unchanged (S3 semantics), so two copies of the same version can differ ONLY in lock state while their whole LWW key ties — and the N1 >= equal-tuple guard passed the tie in BOTH directions: whichever copy was applied LAST won. Reproduced and confirmed at the bit level (rowid/seq-counter forensics on the preserved DBs): node 2 serves the retention PUT while node 3 is down (lock row stamped, audit-confirmed); at phase-D re-entry node 2 restarts, its in-memory HWM resets to 0, it re-pulls node 3's FULL manifest and re-applies the STALE lock-free copy over its own newer lock state (DELETE+INSERT, rowid reused = max+1, fresh seq); the fresh seq then propagates the regression — node 1 sees the clobbered row as "new" and loses its copy too. Final state: retention silently GONE on all three nodes (data loss of WORM state). No apply ever failed, so the M1 skip could never have fixed it. Fix: a dedicated LWW dimension for the lock state — new objects.lock_updated_at column (sqlite migration v23 / pg 0011, NULL = never changed; ObjectRecord.lock_updated_at serde-default → H10-additive on the wire), stamped by set_object_retention/set_object_legal_hold on both backends; apply_remote_object guards become strictly newer last_modified, OR tie + strictly newer lock_updated_at (versioned branch; the null-version register adds it as the final tiebreak after blob_id, whose own >= had the same bidirectional hole). Option ordering (None < Some) makes any post-upgrade lock change beat a never-locked copy; same_replicated_content includes the new field automatically (M7 no-op intact). Unit tests: stale-copy-cannot-clobber (versioned + null-version), lock-only change still applies, both mutations stamp. The phase-D retention integration test is the e2e regression test — it caught the bug.
Console topology card, minimal per plan: an amber "syncing" notice (exact replica of the config-drift warning pattern) + per-node amber sync lag N / N skipped lines (shown only when non-zero) in the node list's right column.

Outcome: a re-entering node serves no wrong answers; the operator sees lag and anomalies; the worker neither stalls nor monopolizes.

R8 — Console: per-node views behind the LB¶

Decision H9 confirmed by Pietro (2026-06-12) on all three axes: server-side proxy; all FOUR node-local view families (the replication journal joins the original three); an "All nodes" merged option in the selector.

D6 (decision H9): a ?node=<node_id> parameter on the four node-local admin endpoint families — /admin/audit, /admin/metrics/history, /admin/notifications/events, and /admin/replication/journal. As built: when present and ≠ self, the query is proxied server-side to the peer's new POST /cluster/v1/admin/{audit,metrics-history,notification-events,replication-journal} receive routes (behind cluster_auth + the R4 mTLS marker, 2 MiB body cap; the filter travels as a JSON body like the manifest request — never a signed query string). The transport is a new ClusterAdminProxy trait in arca-core implemented over ClusterClient in arca-server (same dependency rationale as RawBlobOps). Only ELIGIBLE peers are valid targets (every H12 gate): unknown node → 404, known-but-ineligible → 503, unreachable → 502, and a peer's own 4xx (e.g. "audit not enabled there") forwards with its original status and message. Every response carries a top-level node label — including the default LB path, so the operator always knows which node answered. The receive handlers reuse the exact local-page functions of the admin handlers (single source of truth), so a forwarded query can never re-proxy (the node field is skip_serializing).
?node=all merged view: the entry node fans the same query out to every eligible peer in parallel (plus itself), merges the rows by timestamp — parsed, not string-compared (sub-second precision would order wrongly) — newest-first, labels each row with its source node_id, and reports a per-source sources array (total, or error for a failed source: a partially-failed fan-out is visible, never silently smaller). Pagination is per-source-page (the merge keeps the newest page-size rows across nodes); the approximation is documented in the API/manual rather than papered over with cross-node cursors. A dead peer is excluded from the fan-out (eligible-only), not reported as an error source.
Console: a node selector in the audit/monitoring/events/replication-journal views via a shared nodeSelectorMixin (console/js/node-selector.js), populated from /admin/cluster (eligible nodes + "All nodes"; hidden entirely when clustering is off; a stale selection falls back to the LB default). Default "This node (via LB)" with the responding node_id always visible as a colored mono badge ("via 1a2b3c4d"); in the merged view every row shows its source-node badge (one stable color per node) plus a "N nodes merged" chip and a red "N failed" chip with per-source errors in the tooltip. Monitoring draws ONE SERIES PER NODE in the merged view (x-axis re-mapped by timestamp instead of sample index, per-node legend). Coherence sweep: detail side panels gained a Node row; Clear All (audit/events/journal) and journal Retry are disabled with an explanatory tooltip while a node is selected — those mutations run on the node serving the request, not the viewed one. Gotcha pinned in the mixin: views spread it into their x-data literal and object spread copies a getter's VALUE at spread time, so the mixin exposes plain methods, never getters. Verified live against the 3-node compose cluster with Playwright (specific node, merged view, dead-node degradation).
Screenshots + console manual: documentation/docs/guide/console.md gained a "Per-node views in a cluster" section (manual text; the screenshot pipeline runs single-node where the selector is hidden, so the automated captures are unchanged); the ?node= API and the per-source-page approximation are documented in ha.md. Integration coverage: new bin/test cluster phase K (8 tests, markers cluster_node_views_full/degraded) — per-node audit trails really come from THAT node, self short-circuit, merged ordering/labels/sources, 404 unknown, all four families proxied, 503 dead target, merged shrink to eligible sources; audit+metrics enabled in the cluster test config (not in the drift fingerprint). 15 new unit tests (selector resolution, merge ordering/truncation/failed-source, peer-error mapping).

Outcome: the operator always knows WHICH node they are looking at and can pick it (or see all of them merged), without depending on round-robin luck.

R9 — Documentation, deploy and closure¶

Two R9 decisions taken with Pietro (2026-06-12): §5.4 leftovers = real integration tests (not doc-only) and D5 write-aware health = IMPLEMENTED (not doc-only) — both options the plan had left open. The final version (H11) is v0.26.0.

Outcome: the system declares exactly what it does, the operator has the runbooks, the debt is tracked, the v0.26.0 release closes the remediation plan.

Finding → milestone traceability¶

Update the Status column as work proceeds: ⬜ to do, 🔧 in progress, ✅ done, ➖ decided not to do (with a note).

Finding	Short description	Milestone	Status
§2.1	Quorum = admission gate, not a write quorum	R1	✅
§2.2	`seq` cursor race on PostgreSQL	R1	✅
§2.3	Deletes before tombstones in the control merge	R1	✅
§2.4	Sequential fan-out	R1	✅
§3.1	No anti-replay window	R4	✅
§3.2	Tombstone GC blind to liveness	R3	✅
§3.3	Workers duplicated on every node	R6	✅ (lifecycle leader-gated on the lowest ELIGIBLE node_id; metrics/retention/notifications not gated by design — node-local state; the replication-worker half of the finding was a wrong premise: its journal is node-local, deliveries were already exactly-once, gating would orphan non-leader entries — confirmed by Pietro 2026-06-12)
§3.4	Cluster endpoints without a body limit	R4	✅
§3.5	Public health exposes disk/fingerprint	R3	✅
§3.6	SSE-C not replicated	R5 (spike) + R9 (doc)	✅ (spike outcome: replication already worked via the cluster-wrapped sidecar write; the read side gained `ClusterSsecBlobStore` synchronous read-repair — no TD-017)
§3.7(A)	Rogue peer receives all new data with no secret (fan-out authenticates no peer)	R3 (H12: peer auth, gate fan-out+quorum) + R4 (mutual TLS)	✅ (R3: challenge-response peer auth gates fan-out, anti-entropy pulls, quorum and min_disk; R4: mutual-TLS second factor — client certs enforced on `/cluster/v1/*`)
§3.7(B)	Secret brute-forceable from public fingerprint; weak secrets allowed	R3 (§3.5 fingerprint off public) + R4 (M5 secret strength)	✅ (R3: fingerprint off the public health; R4: M5 floor + placeholder rejection + entropy warning)
§3.7(C)	Plain-HTTP/unverified-TLS/no-replay enable sniff/MITM/replay	R4 (TD-015 verified TLS + §3.1 anti-replay)	✅ (verified mTLS mandatory over HTTPS; ±15 min replay window; plain-HTTP remains an explicit operator choice documented for trusted segments only)
TD-015	Inter-node TLS accepts invalid certs (now committed, not deferred)	R4	✅
M1	HWM stuck on a failing entry	R7	✅ (skip after 5 consecutive failures on the same seq, warning + per-peer `skipped_entries` in `/admin/cluster`)
M2	Repair without a budget	R7	✅ (`[cluster] blob_repair_budget` default 100 fetches/tick + resume cursor; mid-flight sweeps continue every tick)
M3	Membership without eviction	R3	✅
M4	503 without Retry-After	R7	✅ (pinned by unit tests on `s3_error_response`: every ServiceUnavailable carries it, non-503s do not; the syncing health 503 sets it explicitly)
M5	1-character secret accepted	R4	✅
M6	Path blob_id not validated	R4	✅
M7	seq churn on identical rows	R5	✅ (identical-row no-op in `apply_remote_object`, both backends — the churn was a perpetual full-table redelivery ping-pong between caught-up nodes)
N1	Lock changes (retention/legal-hold) don't bump `seq` → invisible to anti-entropy (found during R1)	R5	✅ (fresh `seq` stamped in both backends' lock UPDATEs + catch-up integration test)
N2	Equal-timestamp LWW clobber erases lock state cluster-wide: the N1 `>=` guard passes the tie in both directions, so a restarted peer's stale full-manifest re-pull silently deletes retention/legal-hold everywhere (found during R7 — the real root cause of the phase-D retention flake)	R7	✅ (`lock_updated_at` LWW dimension on both backends — sqlite v23 / pg 0011 — stamped by lock mutations, strict tiebreak in `apply_remote_object`; unit + phase-D integration regression)
M8	Silent mtime fallback	R7	✅ (warning on the fallback, naming the file and the never-reclaims consequence)
§5.1	No available-mode test	R2	✅
§5.2	No control-plane catch-up test	R2	✅
§5.3	Latent flakiness (wait/phase E)	R2	✅
§5.4	Listed test debt (GC, repair, bootstrap, skew)	R2 (bootstrap in the R7 test) + R9 (GC + repair tests; skew = doc)	✅ (R9 phase L: proactive blob repair proven volume-side without a masking GET + §3.2 GC guard blocked/no-resurrection/release, on a gc overlay with the new `tombstone_grace_seconds` knob; clock skew declared a test gap in `ha.md` — Pietro chose real tests over doc-only, 2026-06-12)
§5-deploy	Inconsistent HAProxy fall/rise; k8s probes	R9	✅ (fall/rise motivated per environment + ha.md snippet aligned to production values; sticky + write-aware backends commented in both cfgs; k8s readiness 5s/2 + liveness moved to `?verbose=1` so a long catch-up cannot get the pod killed)
§5-doc1/2/3	Available LWW, runbooks, SSE-C/quorum	R9	✅ (LWW loser-discarded-silently sentence + RPO in the available bullet; six operational runbooks; SSE-C and quorum semantics were already documented by R5/R1)
D1	Ghost quorum (unauthenticated liveness, drift ignored)	R3	✅
D2	No syncing state (404s/partial listings at re-entry)	R7	✅ (first-pass readiness gate on `/admin/health`, per-peer sync detail in `/admin/cluster`, console notice, phase J integration test)
D3a	No nodes > cluster_size guard	R3	✅
D3b	Secret rotation without dual-secret	R4 (+ runbook R9)	✅ (dual-secret shipped + `ha.md` rotation section; R9 consolidates the runbooks)
D3c	Restore from backup: seq rewind vs HWM	R7 (+ runbook R9)	✅ (ping `max_seq` → `PeerNode`, `sync_rewound()` with a ping-fresher-than-last-advance guard, automatic HWM reset; restore runbook stays in R9)
D4	Multipart without reconcile; local-only concat	R5	✅ (uploads + parts in the snapshot, `multipart` tombstone on Complete/Abort, concat repairs missing parts from peers; integration: abort-while-down closes everywhere, Complete succeeds on the returned node)
D5	LB blind to writability (client-visible 503s)	R9	✅ IMPLEMENTED (Pietro, 2026-06-12): `?writable=1` → 503 `read_only` while the write gate is closed, commented write-pool backends in both cfgs, cost+mitigation documented in `ha.md`; integration asserts in phases A and C
D6	Console incoherent on per-node views	R8	✅ (`?node=` server-side proxy on all four node-local families + `all` merged view with per-row source labels; console selector in the four views, mutations gated off proxied views; 8 integration + 15 unit tests)
D7	Quorum reads weaker than the CP label	R9 (doc + sticky reference)	✅ (CP = no conflicting writes, not read linearizability; partition-long staleness on a minority node stated; sticky `balance source` commented in both cfgs and named the reference in `ha.md`)
D8	No real-partition test	R2	✅
D9	TD-016 underestimated (RBAC/bucket_config)	R5	✅ (all 6 families in the snapshot merge with LWW timestamps + tombstones; parent-dead filtering instead of cascade tombstones; TD-016 resolved)
D10	WORM in cluster: trust model undocumented	R9	✅ (dedicated `ha.md` section: verbatim remote applies, secret/CA/OS-access surface, compliance posture recommendations)
D11	Available RPO undeclared	R9	✅ (single-copy durability window named RPO > 0 in the available bullet, with the when-to-choose-it guidance)
D12.1	Receive side without a node-local key filter	R4	✅
D12.2	Export/import rewrites node_id	R5	✅ (export omits node-local keys, import refuses them — double defense + integration test)
D12.3	Failure detector without tolerance	R3	✅
D12.4	Non-empty single-node merge undocumented	R9	✅ (runbook: union+LWW with silent losers, same-master-key prerequisite, discouraged + survivor alternative)

Estimate and sequence¶

R1 ≈ 2-3 d · R2 ≈ 1-2 d · R3 ≈ 1-2 d · R4 ≈ 1 d · R5 ≈ 2-3 d · R6 ≈ 1 d · R7 ≈ 1-2 d · R8 ≈ 1 d · R9 ≈ 1 d → ~11-16 effective days. The R1→R9 order is binding only where there is a technical dependency (R2 verifies R1; R3 provides the ping used by R7; the rest can be reordered if needed).

Out of scope (explicitly deferred)¶

Read-quorum / linearizable reads (stays future hardening, as per the Phase 29 plan).
HLC instead of NTP+LWW.
~~A write-aware health check~~ — implemented in R9 (?writable=1, decision of 2026-06-12); what stays out of scope is only making it the LB default (the shipped configs keep the read-friendly default check, write pool commented).
Automated clock-skew tests (documented as a limit, the NTP constraint is already in ha.md).
Erasure coding, sharding, gossip: out of scope as per the Phase 29 plan.