# Replay and recovery review — 2026-10-11 Review base: `7c7dd4dfb93e4562f8faceb43551afe59c0f6606`. Session: `codex-review-replay-1011`, in its own worktree. Parallel agents reviewed C++, Java and Rust; the root reviewed .NET and serialized builds/tests with host leases and process/memory guards: .NET baselines on BEELINK2 at most 16 build jobs; C++/Rust on HP at most 8 for the first focused baseline and 4 for later checks. The integrated Rust suites and Clippy ran locally. The full C++ suite ran on HP; final .NET validation used SDK 8.0.423 locally after Rust, with each host running only one heavy job. Java review found no new confirmed defect in its replay implementation. ## Findings - **.NET stale finalization:** Close retained the previous connection's acknowledged FinishedSending. Terminate after a later Negotiate, before Establish, could retire the later session ID. Close now resets the handshake while retaining retired IDs, replay history and the outbound frontier. - **.NET failed replay reset:** a failed durable reset returned StoreError while the store retained ownership. The required-failure retirement path now releases the store and stops liveness. Tests model a faulted open journal and a removed mapping; they are not physical disk-failure tests. - **.NET exhausted admission:** recovered exhaustion rejects negotiation with ConfigError. TCP single and direct multi owners now release a rejected child; their TCP multi wrapper closes its corresponding transport. Healthy preflight errors and sibling sessions retain their previous behavior. Exhaustion is the mmap/custom-provider zero-successor convention, not a file-journal claim. - **.NET recovery liveness:** complete automatic recovery requests now refresh outbound silence time. A request at 600 ms moves a 1,000 ms heartbeat deadline to 1,600 ms, avoiding premature traffic; payload and sequence state are unchanged. - **.NET failed required response:** a zero-byte or failed automatic recovery-request write consumed input and left the direct runner live without a retry queue. All incomplete required session responses now use the existing retirement path. Explicit application-send retry behavior is unchanged. - **.NET callback continuation:** an application callback could replace a peer, then return an error that closed its replacement; a callback Terminate could also be followed by FinishedReceiving on the same transport. Lifecycle generation checks preserve the replacement's state, and terminal callback operations stop finalization continuation while normal callback sends still complete recovery. - **C++ high administrative frames:** a high-sequence Reject could lose both admin and rejection callbacks permanently. High Heartbeat/Reject also bypassed selected validation. A subsequent extension reproduced missing TestRequest observation: only its ID survived for a deferred Heartbeat reply. These frames are validated and observed while the original bytes remain owned, with recovery state preserved and callback stop/reset guarded. The application hot path gains no extra work and no replay buffer is added. - **Rust retained-frame observation:** a high application copied back for delivery after gap recovery was counted and audited as another wire receipt. A local re-entry marker suppresses only that duplicate observation. Validation, delivery, actual duplicate wire receipts and per-processing-pass latency sampling remain. The traffic and audit reductions are functional effects, not measured latency claims. No application-send allocation or lock is added. - **C++ test scheduling:** the queue concurrency test required 5,000 handoffs while both threads could busy-spin on one isolated CPU. The unchanged binary hit its 20-second deadline on CPU 8 (20.137 s) and passed on CPUs 0–7 (0.033 s), with no lost or corrupt accepted messages. A test-only backoff when no progress is possible gives its peer CPU time; the deadline, message count and FIFO assertions stay. The fixed binary passed both masks (0.615 s and 0.565 s respectively). - **C++ build dependencies:** generated `.d` files entered `MAKEFILE_LIST`, making unrelated object and direct-source targets depend on each dependency-file refresh. Rules now exclude the exact generated paths from configuration prerequisites, while preserving source/header edges and actual makefiles. Seventy-four dry runs cover every changed rule in both editions. One-object rebuild scenarios triggered two unrelated open or ten unrelated development recompiles before, and zero after. Source/header/configuration changes still rebuild their expected targets, including a real configuration named `.d`. These checks isolate the dependency graph with the existing build-config stamp held old; compile recipes and the stamp rule are unchanged. This is build-graph evidence, not an elapsed build-time benchmark. ## Verification The initial regression fixtures were run against the original runtime sources. C++ then added a second, 69-case fixture to the corrected Heartbeat/Reject runtime to isolate TestRequest observation. Each phase retains its own unchanged fixture for before/after runs, with exact pass, failure and skip identities. A separate .NET callback extension adds 12 cases against the first five corrected runtime paths before adding callback lifecycle protection. | Gate | Result | |---|---| | .NET original 72 cases | 30 expected failures, 42 passes, zero skips; exact failure identities checked | | .NET callback extension baseline | 12 cases: 6 expected failures, 6 passes, zero skips; exact identities checked | | .NET final focused 84 cases | 84 passed, zero failures or skips | | C++ original 46 cases | 34 expected failures, 12 passes; all 106 assertion failures matched | | C++ Heartbeat/Reject focused | 46 passed, zero failures | | C++ TestRequest extension baseline | 69 cases: 13 expected failures, 56 passes; all 56 assertion failures matched | | C++ TestRequest final / ASan+UBSan | 69 passed in each, zero failures; leak detection enabled | | Rust original 16 cases | 10 expected failures, 6 passes, zero ignores; exact identities checked | | Rust final integrated full default / no-default | 2,011 passed in each mode, zero failures; one ignored `builders.rs` documentation example in each. All 16 new replay regressions and all 22 upstream ownership cases passed in each mode | | Rust Clippy | Passed: all targets, warnings denied | | .NET development / open SDK 8 | Development: 2,464 passed, zero failures, two skips (bypass NIC unavailable, separately covered native pool case). Open: 2,238 passed, zero failures, one separately covered native-pool skip; first execution passed | | Both native .NET builds and pool supplements | Both libraries built with 209 exports; both three-case pool supplements passed with zero skips. Native-required guards and copied-library hashes verified | | C++ full open edition first run | 169 ran: 168 successful exits; TX queue concurrency test hit its progress deadline. Two binaries reported multicast loopback skips. Layering and all simulator probes passed. Original failure retained | | C++ bounded rerun after queue fixture correction | All 169 existing binaries and five layering/simulator probes passed. Two multicast skips remain. All 838 source-input hashes and 172 executable hashes stayed unchanged during this run | | C++ build graph | All 74 dry runs passed, covering every changed rule in open and development editions | | Latest upstream imported-admin/config additions | C++ file-store executable passed, including 56 imported-admin patterns; Rust config unit and two imported-admin tests passed in each feature mode; .NET 113 passed in each edition with zero skips, including the native-required guard | | Final .NET classic/FIXP integration | Development 269 passed; open 264 passed; zero failures or skips in either, including the required-native guard | | Nine required .NET publication guard stages | Nine ran and passed; selector reports PARTIAL (exit 2) because only requested stages run | | Export scan of changed engine files | Final scan: 17 files, zero findings | .NET cases include real loopback single/multi acceptors, direct runners, failed writes, exact recovery bytes, replay retention and standby/error/reopen controls. Rust uses public transcript APIs and checks exact inbound audit bytes and order, GapFill and actual application replay, coalesced tails, duplicate wire frames, superseded retained frames and deferred rejection. C++ checks fast/configured/deep validation, gap opening and active recovery, callback stop/reset and retained application/chunk state. These are functional checks, not live NIC benchmarks. The first full .NET development run reported 12 Unix-socket path-length failures from an overly long test-driver `TMPDIR`, one stale finalization assertion, and two socket timing failures. The two timing failures passed in a nine-case isolated rerun; only the stale assertion failed again. The existing finalization test now checks the retained retired ID after closure, rejects that ID on a fresh TCP transport, and admits a different ID. The complete rerun uses a shorter per-session temporary path and serial xUnit collections through a recorded external runsettings file. The two socket timing tests retain their original deadlines and assertions. The finalization fixture replaces its iteration-count poll with a five-second deadline while checking retired-ID rejection and fresh-ID admission. ## Publication and evidence Final integration base: `24a4b50faeb696090a32f281501048c3e6f815d4`. Candidate: `d3b367256cda5940f24db6627ebd6cd3eece017a`. Fix commits: `714823cc3` (.NET recovery), `35fc3ee28` (C++ Heartbeat/Reject), `b1b3a1a4d` (Rust observation), `d9fe6400a` (C++ TestRequest), `bd9975a92` (.NET callback lifecycle), `019c4702c` (.NET retired-ID test), `ae3985d90` (C++ dependency prerequisites), `d3b367256` (queue scheduling test). The evidence distinguishes exact source pins: - `cca927c9d`: C++ 69-case positive and sanitizer runs, plus the first Rust full runs (1,980 passes per mode). The reviewed `SessionProtocol.h` and 69-case fixture stay identical through final integration. Upstream subsequently changed settings/schedule headers; the focused fixture does not parse configuration files or schedules, and its sanitizer result remains attributed to this earlier pin. - `40c6ac6badba318be0c3d218f11a139211281675`: first full open C++ gate, including the queue scheduling failure. Its attempted official retry was stopped during an unnecessary dependency-driven rebuild; no successful official retry is claimed. - `11e7db23319ba3768ccc72f27039f9d4cc0931dc`: test-only queue correction, followed by the audited no-build rerun of 169 binaries and five probes. Source and executable hashes were recorded before and after. Multicast loopback did not deliver on this host, so those two network paths remain unexercised. - `aaacba0f67a926627ba2cd11e8eac056af3a088b`: integrated Rust full runs (1,985 passes per mode) and Clippy, including earlier upstream configuration and TLS changes. OpenSSL-backed tests explicitly required the API to load. - `9747229be`: final .NET suites and pool/publication supplements after the retired-ID fixture correction. Both native artifacts, their test copies, and serial runsettings have recorded hashes. - `0ecb3b961`: all targeted latest-upstream checks passed. Earlier upstream additions change imported-admin fixtures and one Rust config unit only; targeted counts complement the earlier full suites and are not a new full-suite claim. An upstream integration added Rust accepted-connection ownership and factory error/unwind handling. `advance()` and `try_resolve_pending()` remain byte-identical across that upstream range. Final full Rust reruns at `b02cc54cc` passed 2,011 tests in each feature mode, with the same one ignored documentation example. Compilation stays at 16 jobs; tests use one thread to match the upstream qualification of immediate descriptor reuse. C++ source/header and .NET trees are unchanged by this final rebase. Upstream also brought Java configuration changes and a C++ threading-document key rename; inventory and generated coverage checks passed. This batch did not rerun a Java runtime gate for those upstream Java changes. All eight fix patches retain identical range-diff entries. The final upstream rebase adds classic .NET sparse-replay indices and a socket-idle fixture collection. It preserves the FIXP implementation, the full Rust tree and the C++ tree. Final .NET integration at `d3b367256` passed 269 development and 264 open cases, zero failures or skips: both editions cover the 84 new FIXP regressions, finalization, imported admin replay, sparse replay/index checks, socket-idle cases, and required native loading. Native binaries were reused only after source and artifact hashes match. Earlier full .NET totals remain tied to `9747229be`, not presented as full suites after these upstream additions. Publication target: private `origin/main`. The archived publication proof records the final commit and remote ref; the review record itself changes no runtime source. Evidence is preserved under `/home/yann/archive/libhft/reviews/2026-10-11-codex-review-replay-1011`, with original sources, fixtures, patches, exact command results, full logs/TRX reports and a git bundle. Temporary worktree and build outputs are removed after publication and archive verification.