The Load Bench Handbook

Fresh installed .NET affinity smoke: 8/8 cells completed. Seven smoke PASS, one SATURATED; all eight startup-affinity checks pass. Kernel, 3,000 warmup slots and 3,000 measured requests/session, one run; multi uses four processes × eight sessions. All completed cells have zero measured shedding and deficits. These are behaviour verdicts, not stable latency measurements.

Cell	Smoke verdict	Measured replies	Startup affinity
dotnet-native/kernel/single	SATURATED	3000/3000	pass
dotnet-pure/kernel/single	PASS	3000/3000	pass
dotnet-native/kernel/mono	PASS	96000/96000	pass
dotnet-pure/kernel/mono	PASS	96000/96000	pass
dotnet-native/kernel/pool	PASS	96000/96000	pass
dotnet-pure/kernel/pool	PASS	96000/96000	pass
dotnet-native/kernel/shard	PASS	96000/96000	pass
dotnet-pure/kernel/shard	PASS	96000/96000	pass

Native single is SATURATED on scheduler slip (p99 199.842 µs versus 20 µs). Its server now preserves rc=-1 / Disconnected; that ABI code conflates EOF and read errors, so it is not proof of a protocol fault or of a clean FIN. Evidence: dotnet-affinity-smoke-20260908-p1vR5e. No smoke latency is imported. Startup read-back is not continuous placement proof.

Historical v3 C++ live-generation smoke: stopped after 2/4 cells. Newly installed C++ binaries include full-width ownership and exact live-generation validation before reading reply timestamps; source-level wraparound and stale-reply regressions pass. Kernel, fixed OPEN 50,000/s aggregate, 3,000 warmup slots and 3,000 measured requests/session, one run. These are behaviour checks of the existing v3 schedule, not replacement latency data.

Cell	Smoke verdict	Measured replies	Startup affinity
cpp/kernel/single	PASS	3000/3000	pass
cpp/kernel/mono	PASS	96000/96000	pass
cpp/kernel/pool	not launched in this sitting	—	—
cpp/kernel/shard	not launched in this sitting	—	—

Evidence: cpp-generation-smoke-20260908-khwl6J, built from commit 0e12640d, installed/local binary hashes matched. No smoke timings are imported. Completed cells have zero measured shedding/deficits. The earlier two short probes in cpp-marker-smoke-20260908-sM5Sm5 predate this generation guard and are not counted as its installed verification. Neither set proves that bad replies occurred historically. This sitting stopped at preflight when v4 source work began. All four C++ shapes now have installed v4 functional checks below; those are separate evidence, not completion of this old sitting. The sustained maximum-throughput sweep remains pending.

Shared-epoch OPEN v4: all 20 kernel functional cells completed; 17 PASS, 3 SATURATED. The opt-in schedule uses exact cell-wide slots and a common prepared warmup/measurement epoch. At 32 sessions, 50,000/s means exactly 1,562.5/session/s. Service and response retain their separate meanings; delayed sends still contribute scheduler/response latency.

Cell	Smoke verdict	Measured replies	Startup affinity
cpp/kernel/single	PASS	3000/3000	pass
cpp/kernel/mono	PASS	96000/96000	pass
dotnet-pure/kernel/single	PASS	3000/3000	pass
dotnet-pure/kernel/mono	PASS	96000/96000	pass
java-pure/kernel/single	SATURATED	3000/3000	pass
java-jni/kernel/single	SATURATED	3000/3000	pass
java-pure/kernel/mono	PASS	96000/96000	pass
java-jni/kernel/mono	PASS	96000/96000	pass
dotnet-native/kernel/mono	PASS	96000/96000	pass
cpp/kernel/pool	PASS	96000/96000	pass
cpp/kernel/shard	PASS	96000/96000	pass
dotnet-pure/kernel/pool	PASS	96000/96000	pass
dotnet-pure/kernel/shard	PASS	96000/96000	pass
dotnet-native/kernel/single	SATURATED	3000/3000	pass
dotnet-native/kernel/pool	PASS	96000/96000	pass
dotnet-native/kernel/shard	PASS	96000/96000	pass
java-pure/kernel/pool	PASS	96000/96000	pass
java-pure/kernel/shard	PASS	96000/96000	pass
java-jni/kernel/pool	PASS	96000/96000	pass
java-jni/kernel/shard	PASS	96000/96000	pass

Evidence: open-v4-diagnostic-20260908-ZEC8gV; installed managed/shared DLL hashes matched the build. One run, 3,000 warmup and 3,000 measured iterations/session. These are functional diagnostics, not replacement full-size latencies. The existing matrix remains v3: no v4 value is silently mixed into it. Java Pure single delivered every reply without measured shedding, but achieved only 41,281/s; scheduler p99 was 12,560.210 µs. .NET Native single delivered 3,000/3,000 at 49,976/s, but its scheduler p99 was 302.058 µs versus a 20 µs slot interval. Java/JNI single delivered 3,000/3,000 at 49,979/s, but scheduler p99 was 49.302 µs. These short-run saturation flags remain visible; neither is marked as a usable latency-at-load result. A native-only staging mismatch was caught before launch; rebuilding the staging tree and reinstalling produced matching native-library hashes before the completed native run.

Java Pure raw replay shows gradually accumulating lateness, not a single 12 ms pause: first/last 300-request windows averaged 38,863/46,283 task starts/s. This is consistent with a short warmup but does not prove a JIT or GC cause. The unchanged-source normal-size test below greatly reduced that delay but still failed the strict qualification gates. The differing Java versus C++/.NET extraction checksums exactly match their respective request-ID ranges, which the contract permits; they do not demonstrate a decoding mismatch.

Raw replay of all 96,000 measured requests in each C++ and .NET Pure mono cell verifies the intended 50,000/s aggregate pacing: actual task-start mean rates were 50,000.13/s and 49,999.86/s; median cell gaps were 19.947 µs and 19.980 µs respectively. Every replayed response equals service plus scheduler delay, with no missing/clamped samples or measured shedding. These are client task-start timestamps, not wire timestamps. Hashed replay reports are retained as pacing-cpp-mono.json and pacing-managed-mono.json in the evidence directory. Every kernel shape/flavour has now passed functional delivery and startup placement; the three single-cell saturation qualifications above remain. See the v4 schedule contract.

Independent raw replay also passes for all ten kernel pool/shard cells: 96,000 measured exchanges each, zero measured deficits/shedding/raw drops/clamps, and exact service/scheduler/response decomposition. Mean client task-start gaps are approximately 20 µs. .NET Native pool/shard retain maximum gaps of 447.756/445.100 µs; a correct mean does not erase individual delays. Reports pacing-*-pool.json and pacing-*-shard.json retain the hashes and distributions. Optional .NET warmup completion counters were not emitted in this batch, so warmup shedding is unreported, not assumed zero.

Normal-size v4 diagnostic: Java Pure/kernel/single completed. 20,000 warmup plus 20,000 measured iterations per run, three runs; 60,000/60,000 measured replies, zero shedding, deficits or stale replies. Service p50/p99/p99.9/max: 33.600/36.782/74.808/163.822 µs. Response: 34.543/61.040/147.366/201.289 µs. Achieved 49,996/s, but scheduler p99 27.581 µs exceeds the 20 µs interval and run stability is FAIL. These are qualified diagnostic values, not a passing capacity claim. Raw pacing replay, parameter provenance and startup pinning read-back all pass. The standalone pinning auditor now validates the exact v4 recipe fields without weakening its placement rules. Evidence: java-pure-single-full and pacing-java-pure-single-full.json in the same sitting directory.

The publication audit independently reconstructs all four service and response values from the raw samples, using the original per-run/session aggregation method. It found an existing percentile-rank discrepancy: C++ floating-point p99.9 arithmetic can select one sample above exact nearest rank. The functional captures above retain their original method. The integer-rank correction has passed C++, Java and .NET tests plus all 147 protocol-audit checks. All five flavours have been rebuilt and installed from runtime commit cde1f0cd; local/installed hashes match. New full-size captures will identify the corrected method explicitly, without relabelling old evidence. The new batch and live-watcher evidence are retained under open-v4-full-20260908-XmhZ7z. All five C++ single transports completed the normal-size recipe and passed the load, raw and startup-pinning gates. Kernel achieved 49,997/s, with service p50/p99 22.507/28.322 µs and response 23.532/30.569 µs. These are GCC 11 measurements. An incoming GCC 14 build-default change triggered the source-freeze guard before publishing the next Java attempt. The five-result sitting is archived. The rebuilt GCC 14.2.1 baseline open-v4-gcc14-20260908-9PgBhs stopped at 2026-09-08 14:10:04 UTC after 33 of 72 supported cells completed: 19 OK, 13 partial and one saturated. Those audited observations are retained in the separate v4 matrix and this handbook. The next cell, Java/JNI + SocketXtreme + mono, aborted during warmup: two clients each sent 4,096 requests across eight logged-on sessions without a reply and hit the stall guard. No measured latency was imported for that attempt. Its transcripts and failed phase audit are retained; the cause is still under investigation, not classified as a capacity limit.

Compiler and tuning baseline: this GCC 14 batch uses the default C++ kernels, without a local tuning profile. During that baseline, the installed JNI and .NET Native libraries both reported tuned_build=0. Existing local SCAN profiles were generated with GCC 11.5, not GCC 14; the opt-in AVX-512 defaults table does not apply to this Broadwell machine. A fresh full local tune completed on 2026-09-08 with /opt/rh/gcc-toolset-14/root/usr/bin/g++, up to 24 adaptive samples per variant and isolated core 19, separately from load benchmarks. Turbo, governor, isolation and machine-load preflights passed. All 12 probe verdicts pass the current robust tuner gates; any_inconclusive=false and recommend_rerun=false. Its profile and raw evidence are retained under gcc14-local-tune-20260908-7U4n1s; existing profiles are untouched. The selected combination also passes the standalone correctness self-test and eight engine-level primitive tests, including 62,314 field-split corpus cases. The separate tuning record links the preserved candidate profile and explains the variance/reporting limitations. Compared with the tracked GCC 11 SCAN profile, the common primitive picks differ only in decimal formatting: clean_arms becomes current. SSE2 integer formatting, mask8_delim integer parsing and disabled structural parsing remain selected. These are primitive findings, not an end-to-end gain. Thirteen existing whole-FIX correctness suites also pass with this exact profile, including 43 conformance groups and 274 malformed-frame cases under four policies. The tuning record retains commands and qualifications, including compiler warnings in a typed-view boundary test. Runtime commit 9443f6a3 has now been rebuilt consistently and staged: 52 local/installed artifact pairs match, and all three Java/JNI / .NET Native filename routes pass cold profile probes with tuned_build=1. The tuning record qualifies one retained, unreferenced Java adapter jar; this is not a claim of exact runtime-tree equality. Fresh live checks precede the next full sweep; existing rows remain the labelled untuned baseline until remeasured. Pure Java and .NET Pure do not consume the C++ kernel profile. No retuned latency result is claimed here.

SocketXtreme repair validated offline: negative completion-poll returns now remain errors through the C++, JNI and .NET Native callers; ordinary peer disconnects remain per-connection events. Connected initiators elect a drainer per verified RX ring. Failed setup, reconnect and move paths preserve exact packet ownership; missing or multiple RX rings explicitly refuse setup. Nine transport fixture groups and the existing move tests pass optimized and address/undefined-behavior sanitizer builds. Thirty caller cases cover fatal versus ordinary errors, native cached borrows and owning-step cleanup; the portable acceptor regression also passes. These are socket-free tests, not proof that the recorded warmup stall is resolved. That repair was rebuilt and staged.

A subsequent positive-batch ownership repair retains every pending accepted socket, routes existing peers once, releases failed-child/orphan packet claims and preserves terminal events accompanying newly accepted data. Thirty-two new scenarios pass optimized and address/undefined-behavior sanitizer builds; the existing nine poll-error groups and 256-slot move regression also pass. Positive ownership refusals do not fabricate negative-poll evidence. Independent source review found no blocker. The combined repairs were rebuilt from 368b1374 and staged at 2026-09-08 16:12:11 UTC with the same GCC 14 profile: 52 enumerated installed pairs match, and all six named native-library routes pass cold profile probes. The unchanged extra adapter jar remains qualified in the tuning record. Fresh live diagnostic results are still required. This does not change general combined-terminal receive semantics. C++ SocketXtreme now also has explicit coordinator-pool and independent-ring shard routes; their first hardware cells remain pending and are not inferred from the earlier single/mono evidence.

First retuned live check — rejected before import: open-v4-gcc14-tuned-check-20260908-k3jQMn stopped at 2026-09-08 15:20:48 UTC in pure Java/VMA/mono, before reaching the native checks. All four clients completed warmup with shedding, then aborted measured run one with 512 unanswered requests each. Later runs were absent, so the phase audit correctly refused publication. That attempt did not replace any matrix row. The live watcher recorded two busy server threads on core 10 during measurement; source and two startup affinity stamps show the blocking accept loop and sole multiplexer both use that core. VMA's latency preset sets VMA_RX_POLL=-1, causing its blocking accept to poll indefinitely. The Java VMA launcher now uses a finite receive-poll budget of 10 for both roles/providers, matching its former client-only setting. Fifteen launcher regressions pass. Shared-ring traffic can still wake the accept loop. No privileged statistics were collected before these failed JVMs exited; their IPv4 launch option alone is not proof of accelerated socket traffic.

Finite-poll retry — offload confirmed, benchmark failed: open-v4-java-vma-rx10-20260908-1x4K1B completed all six phases at 2026-09-08 15:36:13 UTC, but still shed scheduled requests. The importer correctly publishes fail with blank latency/rate fields; 32/32 records connected sessions, not full delivery. This replaces the old fallback mono observation, leaving 32 numeric v4 observations; original evidence is retained. Prearmed, PID-verified VMA readers matched all 32 IPv4 connection tuples in both directions. All 64 endpoint socket series have increasing transmit and receive offload counters, and final socket error counters are zero. All five collectors exited successfully. Root-owned namespace/FD links remained inaccessible; offload attribution uses the per-process socket tuples and counters, not merely preload, global rings or kernel socket tables. Readers ran on housekeeping cores; no claim of zero diagnostic overhead is made.

The retry still has two active application threads on server core 10 during measurement: watcher sample 19 records +99/+108 CPU ticks, and retained actual task identities name both libhft-pure-jav. Linux name truncation does not independently identify which TID is the acceptor versus multiplexer. The separate VMA maintenance threads remain on reserved core 14. Finite polling did not fix this resource contention. A plaintext mono repair now uses one owner for nonblocking admission and session processing, with one accept attempt per data sweep and no extra core. It keeps accepting/rejecting at the session limit and preserves the separate pool/TLS paths. Seventeen new socket-free tests and 24 existing lifecycle, pool, TLS and reconnect regressions report pass. The repair is now rebuilt/staged as recorded above; a fresh live check is still required. Neither offload nor completed phase files establish clean latency or maximum throughput.

Latest full-size ownership-repair results (2026-09-08): After the ownership controls, the shared-epoch v4 matrix contains 36 numeric cells. The latest four published mono cells are .NET Pure/kernel (OK), .NET Native/SocketXtreme (partial: instability), JNI/SocketXtreme (partial: instability), and pure Java/VMA (partial: instability). open-v4-owner-check-20260908-wWhlDi/sitting-managed-kernel completed at 2026-09-08 16:21:22 UTC. .NET Pure/kernel/mono passes the standard 50,000/s aggregate, 32-session v4 recipe: all four clients complete all six phases with zero shedding, send refusals or outstanding replies. Service p50/p99 is 147.30/191.94 µs; response is 149.609/199.114 µs. The result and startup-affinity audits pass. A prearmed read-only observer retains 65 snapshots of the actual five CLR processes, including named server-GC workers on cores 0–9. The live watcher reports no application collisions in its bounded samples; sleeping socket/signal helper threads still share some application cores. This is target-throughput evidence, not a maximum-rate result.

The same build's .NET Native/SocketXtreme/mono control completed at 2026-09-08 16:24:46 UTC under sitting-native-sx. All 32 sessions completed the scheduled load, with no phase shedding, send refusal or outstanding replies. Response p50/p99 is 13.868/21.287 µs. The matrix retains it as partial for run-to-run latency instability, not as missing-reply loss. The prearmed CLR observer completed 73 snapshots of all five process identities; sampled native owner/client cores are 10 and 11/12/13/15, with VMA helpers on 14. JNI/SocketXtreme/mono also completed all 1,920,000 measured exchanges at 2026-09-08 16:28:09 UTC under sitting-jni-sx, with all 24 cold phase records reporting complete delivery and zero shedding/refusal/outstanding. Response p50/p99 is 13.567/20.983 µs, again qualified partial for latency instability. The earlier warmup stall did not recur in this fresh sitting; one successful control does not prove the original failure's cause.

Earlier Java Pure/VMA ownership control — publication withheld: sitting-java-vma completed all 1,920,000 measured requests with zero measured shedding or outstanding replies, but the startup audit still expected two server affinity stamps and rejected the new one-owner layout. Logged service p50/p99/p99.9/max is 17.642/22.187/25.039/436.218 µs; response is 18.664/23.976/28.051/438.514 µs. These are provisional diagnostic observations with run-to-run instability, not an admitted matrix cell. Its observations were not imported. A new explicit cold topology marker and corresponding strict audit now pass 18 Java startup tests and 30 placement tests. Java/JNI was rebuilt from e72c35d8 and staged at 2026-09-08 17:03:44 UTC: all 52 enumerated installed pairs match and all three JNI library routes pass the cold tuned-profile probe. C++ and .NET Native retain the previous verified build. The independent thread observer retained 19 measured-phase snapshots: exactly one application task on server core 10, with matching identity and approximately one CPU's accounted work. VMA maintenance remains on 14. Initial warmup shed 37,147 scheduled requests; later warmups and all measured phases completed fully.

Per-socket VMA statistics confirm increasing RX/TX offload for all 32 server endpoints and 24 client endpoints, forming 24 fully paired connections with zero observed final socket errors. Client three's reader started before its statistics were available and returned no data, despite exit code zero. Therefore this attempt does not claim all 64 endpoint counters and is not retroactively upgraded.

Pure-Java/VMA ownership retry — published with instability qualification: sitting-java-vma-marker completed at 2026-09-08 17:07:54 UTC. All 32 sessions delivered 1,920,000 measured replies across three runs, with zero measured shedding, deficits, stale replies, send refusals or outstanding requests. The explicit fused-mono marker and actual affinity read-back pass. Service p50/p99/p99.9/max is 17.365/21.888/27.382/450.424 µs; response is 18.477/23.877/29.670/451.971 µs. The row is partial because two clients report run-to-run instability, not because replies are missing. First warmup shed 45,599 scheduled slots but drained every actual send; later warmups and all measured phases completed without shedding.

Readiness-gated, identity-checked VMA readers observed increasing RX/TX offload at all 64 endpoints, covering all 32 reversed connection tuples, with zero observed final socket errors. All five readers completed. This verifies offload, not wire timestamps, observer neutrality or maximum throughput.

Temporary Mellanox-only campaign (2026-09-08): While the Solarflare DAC is removed, KERNEL_NIC=mlx selects ordinary kernel TCP over mlxA/mlxB, alongside VMA and SocketXtreme on the same Mellanox ports. Onload and TCPDirect are excluded. The fresh open-v4-mellanox-20260908-PekVA8/sitting selects 46 original-flavour cells at the unchanged 50,000/s aggregate v4 target (1,562.5/s per session at 32 sessions), publishing after each completed cell. Earlier kernel rows used Solarflare and remain historical until individually remeasured; the campaign's completed-cell list identifies the replacement rows. Its retained machine configuration and recipes record the NIC-path change. These are target-throughput results, not a maximum-throughput sweep.

Initial Mellanox single-session findings: .NET Native/kernel completed all 60,000 measured requests, with no shedding, refusal or outstanding requests at phase end. Its saturated flag comes from scheduler p99 153.189 µs exceeding the 20 µs interval, despite approximately 50,000/s achieved. Exact raw replay confirms per-run scheduler p99 values of 424.817, 15.647 and 19.102 µs; service p99 stays between 44.392 and 44.798 µs. Run 1 drives the scheduling qualification, not a percentile-calculation error. This is a failed offered-load timing gate, not proof of server-capacity exhaustion; the retained logs do not identify the cause of the client scheduling pauses.

Fresh .NET Native/kernel single control (22:40:34 UTC): the corrected installed build completed all 60,000 measured requests at 49,997/s. All six warmup/measured phases drained without shedding, refusal or outstanding requests. Scheduler p99 is now 7.380 µs, below the unchanged 20 µs gate: this run does not reproduce saturation. Service p50/p99/p99.9/max is 22.794/31.102/43.153/250.078 µs; response is 24.767/33.747/175.488/264.483 µs. It remains partial for a service-p50 stability warning: run medians 24.394/21.927/22.060 µs give CV 6.0873%, above the unchanged 5% warning limit. An early scheduling transient remains (65 requests above one interval in run one), but not the earlier 617-request episode. The new archive preserves the separate observation; multiple source corrections separate it from the old row, so this is not a one-factor causal comparison.

Java-pure/VMA single failed: 53,127 of 60,000 measured offers were shed at the client in-flight cap; all 6,873 actual sends completed and every phase drained. That attempt's classic path had a blocking listener and separate reader on core 10, contrary to SINGLE's one-owner contract. Live-watch samples 28 and 29 also report two busy application threads there; exact Java thread roles were not captured. VMA contention is a hypothesis, not a proven transport failure. That failed attempt had blank latency fields; the fresh repair below supersedes it.

Java SINGLE repair, remeasured at the fixed target: commit 2f28abbe adds an explicit pure-Java classic single-owner mode: the listener serves its one connection inline, then resumes acceptance. It does not relabel the multiplexed model or change ordinary maxSessions=1 configurations. The launcher and pinning audit require the matching mode, profile, owner marker and affinity read-back. Focused configuration, lifecycle, TLS and launcher tests pass; the protocol audit passes all 147 checks. Matching GCC 14 Java/JNI artifacts and launcher were built, installed and hash-verified before the fresh open-v4-mellanox-followup-20260908-8JscCC/java-single sitting. Kernel completed at 19:06:22 UTC and VMA at 19:07:22 UTC on 2026-09-08. Each delivered all 60,000 measured replies and all 60,000 warmup replies, with zero shedding, deficits, send refusals, stale replies or outstanding requests.

Kernel service p50/p99/p99.9/max is 31.588/35.790/73.214/154.224 µs; VMA is 9.225/10.701/17.296/113.571 µs. Response is respectively 33.281/46.266/118.401/181.162 µs and 9.506/11.180/39.819/123.223 µs. Achieved rates are 49,996/s and 49,999/s against the unchanged 50,000/s target. Both rows remain partial for run-to-run latency instability, not missing replies. Their one-owner/profile/affinity and parameter audits pass; VMA maintenance threads read back core 14. The live watcher recorded 11 PASS, four INCONCLUSIVE and zero VIOLATION samples. These discrete observations do not prove continuous placement, offloaded packet counters or maximum capacity.

Mellanox sweep stopped after 20 completed cells: the next JNI/SocketXtreme mono attempt, attempt-b1wplnoh, aborted during its first warmup. The first three eight-session clients reported no execution reports; the fourth made progress and drained its actual sends. No measured phase completed. The parameter/provenance audit correctly refused the aborted phase evidence before import; any retained numeric value for this combination is from an earlier attempt, not this failure. Its cause remains under review.

JNI/SocketXtreme localization diagnostic (19:11 UTC): jni-sx-first-split-live-20260908-DQV3oW completed 128,000 warmup and 128,000 measured exchanges across 32 sessions, with zero shedding or outstanding replies. All 32 native server slots advanced inbound and outbound sequences to 8,002 before connection cleanup. The earlier stall did not recur. This shorter, one-run experiment enabled per-read tracing and therefore contributes no latency row and does not establish that the intermittent fault is fixed. The 25 other pending Mellanox cells run independently of this combination.

Mellanox follow-up progress (19:24 UTC): the first five cells in open-v4-mellanox-followup-20260908-8JscCC/remaining completed: .NET Native kernel/VMA/SocketXtreme mono and .NET Pure kernel/VMA mono. Each delivered all 1,920,000 measured replies, with zero measured shedding, deficits or send refusals. Native VMA/SocketXtreme remain qualified partial for variance; the other three are ok. First-run warmup shed 3,777 offers for native VMA and 77,032 for managed VMA; every actual send was answered and every phase drained. Parameter and startup pinning audits passed, including VMA maintenance placement on core 14.

C++ pool live-contention qualification (2026-09-08): the kernel and VMA pool cells completed at 19:27:10 and 19:29:39 UTC, respectively, but their live audit observed the server's own accept coordinator consuming CPU alongside worker 0 on core 10 during all three measured phases. In the VMA run these were main TID 3704243 and worker TID 3704253 in the same process; this was not a stray server or the VMA maintenance thread, which is assigned core 14. The old harness passed --core 10 --shard-cores 10,11,12,13; the C++ pool pins its calling coordinator to the scalar core and separately pins its workers to that list. The coordinator sleeps for 1 ms after an empty accept, but its recurring wakeups still contend with worker 0. Positive CPU ticks establish activity, not an infinite busy-spin or a quantified latency penalty. Both archived rows retain their real measurements and existing partial stability status, with this additional placement qualification; neither is an uncontended baseline. No runtime placement was changed during that frozen sitting. The separate correction and replacement measurements below do not retrospectively certify the old data.

Coordinator correction and C++ rerun: SCAN now declares a pool-only coordinator reservation on isolated core 19, separate from workers 10–13, clients 15–18 and VMA maintenance on 14. C++, Java/JNI and both .NET pool harnesses and actual-role checks have been updated. Java's event drainer and .NET Native's parked control thread have separate, verified housekeeping placement; they are not serving workers. Focused configuration, runtime and affinity tests pass. The C++ binary, launcher and machine configuration are now rebuilt/staged with matching hashes. The replacement kernel pool completed at 20:24:51 UTC: 32/32 sessions, service p50/p99 30.153/39.544 µs, complete samples, and verified coordinator/worker/client separation. It remains partial for stability, not placement or delivery. VMA pool completed at 20:27:20 UTC: 32/32 sessions, service p50/p99 11.137/25.033 µs, complete samples, verified role separation and the same partial stability qualification. The live watcher recorded 28 PASS, 15 INCONCLUSIVE startup/teardown/no-application samples, and zero VIOLATION or INVALID samples. These are bounded observations, not a continuous audit. The two-cell rerun archive preserves both new results and evidence locations. Matching Java/JNI and .NET production builds have now passed at source checkpoint 7610e814 and are staged. Java-pure/VMA pool completed at 21:06:35 UTC: all 32 sessions and all 1,920,000 measured replies, with zero shedding, refused sends or outstanding requests in any phase. The old run shed 858,588 measured offers and had independently observed core-10 contention. The replacement service p50/p99 is 17.512/20.690 µs; it retains its real partial stability qualification. See the Java pool rerun archive. Other corrected provider pool reruns remain pending. Startup identity checks supplement, not replace, the live CPU audit.

Partial/unstable investigation: replaying the retained raw samples for the two old C++ pool cells, Java-pure kernel/VMA single and managed .NET/VMA pool reproduced all 98 session classifications. All samples completed; the qualification reflects latency variation, not missing replies. The old C++ cells have four p99 warnings and two p50 warnings respectively. Both Java single cells have a substantially higher first-run p99.9; its cause remains under investigation. A separate GC diagnostic retained all four logs without changing the matrix. A regression also reproduced Java v4 warmup skipping the sample stores used during measurement. The correction warms and discards those scratch records, preserving measured-only raw data, sinks and counters. The 53-test MultiClient/barrier suite passes. Matching production artifacts are rebuilt, staged and verified. The uninstrumented kernel/VMA single reruns completed at 21:22:46/21:23:46 UTC with all 60,000 measured replies per cell and no deficit; both remain partial for instability. The correction has not eliminated the remaining variance. Their separate result archive preserves the new evidence. See the raw stability review. The fresh coordinator reruns also match all 64 raw-derived session classifications: 30 PASS and two WARN sessions per cell. Kernel's warnings are p99 CV 13.86%/13.61%; VMA's are p50 CV 5.16%/5.89%. This rules out a classification error in those results, not every possible timing defect. Thresholds and measured-run inclusion are unchanged.

Legacy C++ idle-read correction: an in-memory regression reproduced a live session being removed after a ready-but-idle nonblocking read. The legacy acceptor now carries explicit EOF separately from “no data yet”, including its manual benchmark loop; real FIN, errors, buffered fragments and fused dispatch retain their distinct handling. The new normal C++11 correctness target and five related suites pass. This is a confirmed code defect, but its occurrence in the earlier VMA shard failure has not been proven. The corrected C++11 production binary is rebuilt/staged. The fresh VMA shard cell completed at 21:02:31 UTC: 32/32 sessions, service p50/p99 10.136/30.794 µs, with the startup affinity audit passing. It was initially published as partial, but its numeric fields are now withdrawn from the current v4 matrix because the transcript does not prove four active listeners. The rerun archive preserves the replacement result without deleting the original failed evidence.

Native diagnostic-copy correction: hftnet now has an additive pre-Start capture option. Existing library users retain their last-inbound debug snapshot by default; .NET Native and Rust benchmark clients explicitly disable that extra copy without disabling raw replies, parsing or extraction. Native compatibility/sanitizer checks and 56 managed tests pass. Rust was source-reviewed but cannot be compiled on SCAN's current toolchain. Matching native/managed builds are now staged; fresh measurements are still required. This does not remove every inbound event-ring copy.

.NET Pure/kernel mono remains repeatably slower (service p99 180.762 µs, response p99 188.483 µs). Source review found that a successful nonblocking read returns before parsing the new bytes, deferring a complete frame to the next peer sweep. The correction now handles one fresh complete frame during the same visit, preserving one-read/one-frame fairness. The old defect was reproduced before the fix; 132 focused managed tests pass afterward. With the corrected binary, kernel mono completed at 21:12:22 UTC: 32/32 and ok, service p50/p99 103.457/177.984 µs. This does not establish a large latency improvement. The subsequent VMA mono attempt failed before measurement: client indices 0, 1 and 2 each logged on all eight sessions, then each sent 4,096 warmup requests without a reply and hit the unchanged watchdog. Client 3 answered all 99,700 actual sends but shed 60,300 warmup slots before the prepared-phase abort. The provenance audit correctly refused publication; the older VMA matrix row remains historical, not a successful rerun of the correction. Evidence is retained under provider-corrections-rerun-20260908-PNfKFd/managed-mono. The new failure is under investigation, not attributed to the change without a discriminating test. All five processes used VMA_RX_POLL=-1. The launcher's historical reason for leaving .NET unbounded described an older per-session client, whereas v4 multiplexes eight sessions per client process. A validated, managed-VMA-client-only --vma-rx-poll flag now permits a nonpublishing 10-versus-−1 comparison. Both probes completed at 21:42:10/21:49:57 UTC with the same installed binaries and workload: each delivered all 1,920,000 measured replies with no measured shedding, refusals or outstanding requests. First warmups shed 68,192/53,863 offers respectively, but every actual send drained. The stall did not reproduce with either setting; this pair does not establish a polling cause or justify changing the default. Normal defaults, the server poll policy, 50,000/s aggregate rate and watchdog remain unchanged. Diagnostic values do not replace canonical matrix rows; the control record preserves the outcome and its limits.

Managed peer retirement: mono, pool and thread-per-session servers previously kept retired sockets open until all siblings finished. Actual-method regressions reproduce this; the correction closes each retired socket promptly, including when optional dump flushing throws, while letting the healthy sibling finish. Cold per-peer records retain the observed error, cleanup failure and whole-session counters. Retirement/“served” counts are explicitly not proof of measured delivery. Existing fault policies remain, including the legacy thread-per-session cleanup exit policy. All 200 managed MultiClient tests pass, including 12 retirement tests. The matching production build is staged and verified. The fresh canonical managed VMA mono result completed at 22:37:43 UTC: ok, 32/32 sessions, all 1,920,000 measured replies and all 32 service stability checks PASS. Service p50/p99/p99.9/max is 20.658/24.991/25.255/51.551 µs. First warmup shed 64,904 offers; every actual warmup send drained, and no measured offers were shed. All 32 cold peer records confirm socket closure without cleanup errors; their whole-session order counts match actual sends. The watcher retained 14 PASS and seven INCONCLUSIVE samples with no violation. The separate archive preserves the result. This success does not prove the cause of the old missing-ER warmup failure.

SocketXtreme shared-ring progress: HP's drainer correction keeps polling for fed peers even when the drainer has unread completions of its own. Integration tests also reproduced a held-partial-packet regression in that change. The follow-up preserves HP's queue compaction while allowing polling during a borrow: the borrowed payload and ownership descriptor live outside the compacted completion queue. All 14 progress/ownership/error test groups pass in optimized and sanitizer builds; existing move and accept-ownership tests also pass. This is code-level validation, not a new NIC result. Matching C++/JNI/.NET production builds completed at 22:08–22:09 UTC and are installed. Verification matches all 47 in-scope installed files, including transport aliases, against the frozen builds. The first canonical rerun, Java/JNI SocketXtreme mono, completed at 22:15:26 UTC: 32/32 sessions, service p50/p99 12.698/19.516 µs. It remains partial for latency instability; all 1,920,000 measured replies arrived, with no shedding in any phase. The raw replay gives 19 PASS / 12 WARN / one FAIL stability classifications: C3S8's service p50 CV is 10.2631%, above the unchanged 10% failure limit. The other warnings concern service p99 variation. The watcher retained 17 PASS and seven no-application INCONCLUSIVE samples, with no violation; sampled placement is not continuous proof. The separate archive preserves its provenance.

.NET Native/SocketXtreme mono rerun (22:23:33 UTC): all 32 sessions completed all 1,920,000 measured replies; every warmup and measured phase drained without shedding or refused sends. Service p50/p99/p99.9/max is 12.343/19.484/23.277/189.144 µs. Raw replay reproduces 10 PASS / 22 WARN / no FAIL service stability classifications: 21 sessions have p99 variation above the warning limit, and two have p50 warnings (one overlaps). No thresholds were relaxed. The watcher retained 15 PASS and eight INCONCLUSIVE samples, with no violation; this is sampled, not continuous, placement evidence. The cell remains partial for latency variation, not missing replies. Its separate archive retains the result and provenance.

C++/SocketXtreme mono rerun (22:32:41 UTC): all 32 sessions completed all 1,920,000 measured replies, with no shedding, refused sends or outstanding requests in any phase. Service p50/p99/p99.9/max is 13.141/22.480/25.107/71.658 µs. Raw replay gives 31 PASS / no WARN / one FAIL stability classification: CLIENT48's p50 runs 15.061/12.305/12.056 µs produce CV 12.6912%, above the unchanged 10% failure boundary. The watcher retained 15 PASS and seven INCONCLUSIVE samples with no violation. The new result remains partial for latency variation. All three native-backed SocketXtreme mono reruns now have full delivery; that is not proof that all intermittent stalls or latency instability have been eliminated.

V4 warmup-recording parity (2026-09-09): C++ and both .NET manual OPEN clients skipped some sample-storage work during warmup. They now prepare and fill discarded service/response/scheduler buffers in warmup too. Only measured phases contribute to published raw samples, latency, elapsed time, overload counts and in-flight peaks; v3 is unchanged. The 213-test .NET suite, C++ synthetic result/recording checks and all 147 protocol checks pass. Source mutation tests reject warmup skips and scratch leakage. Code generation also caught a non-inlined .NET Native helper; its stores remain directly in the loop, with no added recorder calls. This repairs a workload mismatch, not a proven cause of every early pause. The C++ and C# production rebuild/install from ec56e941 is now verified. The .NET Native library and Java binaries remain unchanged.

Fresh .NET Native/kernel SINGLE (2026-09-09 22:06:57 UTC): the warmup-recording control delivers all 60,000 measured replies at 49,997/s, with no shedding or native errors. Service p50/p99/p99.9/max is 23.349 / 30.595 / 38.144 / 76.555 µs; response is 25.278 / 33.279 / 41.206 / 78.474 µs. It remains partial for service p50 CV 6.1454% (runs 25.000/22.406/22.640 µs), not saturation: scheduler p99 is 7.411 µs, below 20 µs. The previous early late cluster is absent, but the first-run median shift remains unexplained; this different-day comparison does not establish causation. Raw replay validates all nine series and exact service + scheduler = response. The watcher retains two PASS and three INCONCLUSIVE samples, no violation; it is sampled, not continuous proof. Prior observations remain archived.

.NET Native/kernel POOL (2026-09-09 22:12:54 UTC): the fresh qualified result delivers all 1,920,000 measured replies: 27 session stability PASS and five p99 WARN, hence partial. Service p50/p99/p99.9/max is 34.278 / 39.902 / 46.206 / 105.988 µs; response is 37.097 / 46.896 / 53.190 / 112.004 µs. The live audit did not pass: 11 VIOLATION and 10 INCONCLUSIVE samples exposed all four native workers inheriting the coordinator's hft-pool-coord OS name. The retained actual-mask snapshot shows coordinator 19, workers 10–13 and control 0–9, not a shared measurement core. The common C++ worker entry now assigns and verifies hft-pool-worker before serving, with a reproducing Linux regression; duplicate-role audit rejection remains strict. Name ambiguity does not explain the p99 variance, and the historical audit failures remain visible.

Worker-name correction verified (2026-09-09 22:26:51 UTC): C++, JNI and .NET Native were rebuilt and staged from 5e011904. The new native/kernel POOL retains 11 live-audit PASS and 10 INCONCLUSIVE samples, no violation, with one coordinator on 19 and explicitly named workers on 10–13. All 1,920,000 measured replies complete, and all 288 raw series replay without drops or clamps. Service p50/p99/p99.9/max is 32.477 / 48.961 / 53.078 / 91.695 µs; response is 40.844 / 56.999 / 62.777 / 104.814 µs. It remains partial: 29 session PASS, two p99 WARN and one p50 FAIL. CLIENT32's persistent 23/32 µs modes change population across runs: its p50 is 31.786/24.263/31.850 µs, while p99 stays 34.065/34.168/34.112 µs. That explains the 14.8875% median CV, not the modes' cause. No samples or thresholds were changed; placement is sampled proof and latency variation remains under investigation.

.NET Native/VMA POOL warmup failed (2026-09-09 22:33:28 UTC): the next attempt produced no measured samples. Clients 1–3 each sent 4,096 requests without a reply and timed out; client 4 completed 85,333 warmup requests, then could not enter measurement after its peers aborted. The audit rejected the incomplete phase evidence; the previous latency row is unchanged, not a new successful result. The live watcher has two PASS and six INCONCLUSIVE samples, no violation. Correct worker identities did not fix this separate stall. Native LoggedOn/zero-error counters alone did not establish progress; no timeout, cap or polling setting was relaxed.

.NET Native/VMA POOL statistics diagnostic (2026-09-09 22:51:55 UTC): the non-publishing follow-up completed all 1,920,000 warmup and 1,920,000 measured requests without shedding, refusal or outstanding replies. Read-only per-process VMA statistics show no observed socket errors, receive drops, completion-queue drops or buffer exhaustion; the live watcher records 13 PASS and three INCONCLUSIVE samples, no violation. This did not reproduce or explain the earlier stall: observers were attached and a different listening port was used, with the same recipe and installed build. Three sessions still report stability WARN. Diagnostic latency values were not imported into the matrix; the failed attempt remains recorded.

.NET Native/VMA POOL normal control (2026-09-09 23:00:28 UTC): the fresh audited run, without statistics observers and back on the failed attempt's port 45702, completed all 1,920,000 warmup and 1,920,000 measured requests. The matrix now contains its service p50/p99/p99.9/max of 15.686 / 29.092 / 35.011 / 57.268 µs, and response 16.541 / 30.156 / 36.794 / 59.512 µs. All 288 raw series replay without drops or clamps; 192 native phase-health records are clean. The live watcher has ten PASS and 13 INCONCLUSIVE samples, no violation. The row remains partial: 28 session PASS, two p50 WARN and two p50 FAIL. The failing medians are 12.672/12.276/14.847 µs (CLIENT22, CV 10.4356%) and 14.988/12.507/14.985 µs (CLIENT42, CV 10.1098%); both sessions' p99 CV is below 0.31%. This isolates the current stability gate to median variation, not lost replies, but does not explain that variation or eliminate the intermittent warmup stall. Native binaries remain the verified 5e011904 build; the current launcher is staged and hashed. No timeout, cap or stability gate changed.

C++/VMA POOL per-thread rings (2026-09-09 23:09:47 UTC): the new full-recipe run verifies VMA_RING_ALLOCATION_LOGIC_RX=20 in all five processes' VMA startup readbacks. This is the newly committed C++ pool default, not a silent relabel of earlier shared-ring data; other providers do not automatically inherit that setting. All 1,920,000 warmup and 1,920,000 measured requests complete, with 288 exact raw series and no drops or clamps. Service p50/p99/p99.9/max is 14.128 / 37.707 / 48.854 / 70.158 µs; response is 14.923 / 38.763 / 50.140 / 70.491 µs. The live watcher has 15 PASS and eight INCONCLUSIVE samples, no violation. The row remains partial: 31 session PASS and one p50 WARN (client process 3, local session 6; medians 12.805/11.315/12.674 µs, CV 6.727%). This run had no long stall, but does not establish that per-thread rings improve SCAN over its previous configuration or explain the remaining median variation.

C++/VMA ring comparison (2026-09-09 23:18:38 UTC): Two separate, non-publishing controls held the C++ binary, port, 50k aggregate rate and core layout fixed. Ring-per-interface (0) completed all 1,920,000 warmup and 1,920,000 measured requests at logged service p50/p99/p99.9/max 11.096 / 24.672 / 30.289 / 48.362 µs. Returning to ring-per-thread (20) also completed everything, at 13.985 / 37.034 / 47.787 / 67.629 µs, close to the earlier ring-20 run. Exact service/response quantiles and logged aggregation replay correctly. All raw series reconcile; ring 0 has one median WARN and ring 20 has two, with no live-pinning violations. The repeat supports a SCAN latency cost associated with this setting; neither control reproduced HP's long stall, so it does not establish shared-ring reliability. These diagnostics do not replace the current matrix row. Evidence is retained under /tmp/libhft-cpp-vma-rings-20260910-dFEZjl/comparison-ring{0,20}.

Java-pure/kernel SHARD (2026-09-09 23:31:58 UTC): the fresh GCC 14 / Java 17 build and installed runtime have been verified after synchronizing HP's fixes. On the temporary Mellanox kernel path, the full v4 cell completes all 32 sessions and passes the normal stability gates. Service p50/p99/p99.9/max is 46.785 / 62.131 / 67.816 / 1972.220 µs; response is 50.549 / 67.392 / 75.195 / 1979.567 µs. The new row is published in the target matrix. Exact raw replay verifies all four service/response quantiles, all 32 session stability passes, full delivery and zero shedding. The live watcher records 12 PASS and eight INCONCLUSIVE samples, with no violation. A stable percentile result does not explain the approximately 2 ms maximum; tail investigation remains separate from these successful evidence checks. Raw localisation finds 129 service samples above 1 ms out of 1,920,000, spread across all four clients and all three runs; the largest individual sample is 2423.730 µs (the matrix reports the existing run-mean maximum aggregation). No scheduler-delay sample exceeds 200 µs. These outliers are therefore inside the service interval, not solely late scheduled starts; their server, network or runtime cause is not yet established. Evidence: /tmp/libhft-java-shards-20260910-ffC4Df/java-pure-kernel-shard.

Java-pure/VMA SHARD placement failure (2026-09-09 23:39:22 UTC): the first one-JVM run failed during warmup and was not imported. All four fused owners actually pinned to core 10: the bench passed a distinct scalar workerCore to each shard but also passed the full shared workerCores array, whose first entry the fused owner selects. The live audit caught four busy threads sharing that core. Each in-process shard now receives only its own core in both config fields; ordinary pools retain their full worker set and separate coordinator. A config-only regression reproduces the old mismatch without opening sockets or loading JNI. The subsequent rebuild and installed-artifact verification passed; the fresh run is recorded below rather than overwriting this failure. Evidence: /tmp/libhft-java-shards-20260910-ffC4Df/java-pure-vma-shard.

Java-pure/VMA SHARD affinity fix verified (2026-09-09 23:47:53 UTC): the rebuilt f71c740f benchmark now has one fused owner on each of cores 10–13, clients on 15–18 and all five VMA maintenance threads on 14. All 32 sessions finish the v4 recipe, with all 1,920,000 measured requests answered and no measured shedding. The first warmup sheds 714 slots (369 in client process 3 and 345 in process 4); every sent warmup request drains. The contract permits this, but it is not a loss-free warmup. Service p50/p99/p99.9/max is 17.785 / 21.943 / 23.615 / 618.293 µs; response is 18.499 / 23.921 / 24.944 / 620.305 µs. This historical row was initially published as partial, but is now withdrawn from the current v4 matrix because four-listener activity was not recorded. Its original stability remains 31 session PASS and one p50 WARN (client process 3, local session 5; medians 16.857/15.487/16.999 µs, CV 5.077%, with p99 CV only 0.179%). Exact replay and phase totals verify all measured samples and quantiles; the live watcher records 16 PASS, eight INCONCLUSIVE samples and no violation. Correct placement and complete measured delivery do not explain that remaining variation or prove the absence of intermittent VMA stalls. Evidence: /tmp/libhft-java-shards-20260910-ffC4Df/java-pure-vma-shard-fixed.

Java/JNI/kernel SHARD (2026-09-09 23:56:27 UTC): the Mellanox kernel run completes all 32 sessions with verified startup placement. Service p50/p99/p99.9/max is 35.430 / 50.186 / 51.832 / 396.823 µs; response is 40.233 / 54.386 / 58.991 / 402.187 µs. The row is published as partial: 27 session PASS, three p50 FAIL and two p99 WARN. Exact replay verifies all 1,920,000 warmup and 1,920,000 measured replies, no shedding and the logged quantiles. The live watcher has 13 PASS, seven INCONCLUSIVE samples, no violation. Distribution review finds two service-latency groups about 8 µs apart in all three median failures: for client 1/session 2, the fraction below 28 µs changes from 49.325% to 45.425% to 66.010%, moving the median between approximately 32 and 24 µs while p99 stays near 34.5 µs. The data explains the gate mechanically, not the cause of the modes. There are eight individual service samples above 200 µs and one above 1 ms (1024.096 µs), versus 299 and 129 in the pure-Java kernel run. No scheduler sample exceeds 200 µs. NIC coalescing remains an unverified hypothesis: reading its current settings requires an additional sudo credential; no NIC setting was changed. This is not a capacity or stability-pass claim. Evidence: /tmp/libhft-java-shards-20260910-ffC4Df/java-jni-kernel-shard.

Java/JNI/VMA SHARD warmup failure (2026-09-10 00:04:05 UTC): the fresh run verified four distinct worker pins on 10–13, clients on 15–18 and VMA maintenance on 14, but failed before measurement. Clients 1–3 each sent 4,096 warmup requests and received zero replies; client 4 completed 131,024, shedding 28,976 slots. The normal 2-second no-progress guard stopped the stalled clients. No latency value was imported. This resembles the earlier native/VMA pool stall and is not explained by the corrected Java shard affinity. A separate, nonpublishing transport-statistics diagnostic follows below; no timeout or gate in the canonical benchmark has been changed.

Java/JNI/VMA SHARD statistics diagnostic (2026-09-10 00:09:58 UTC): the nonpublishing follow-up reproduced the stall despite a diagnostic-only 60-second no-progress window. Clients 1–4 completed only 10/0/0/20 warmup replies, each retaining 4,096 unanswered requests. All 32 client/server endpoint pairs match in the VMA statistics: 29 server sockets show only their initial logon receive, three show ten additional receives; every client socket reports 32–48 retransmissions. This points below FIX reply processing, but does not identify the failed transport stage. Sampled VMA drop/buffer counters are zero, not proof of no packet loss. Terminal warnings damaged some records; those socket records were excluded, not repaired. Live affinity has 12 PASS, two INCONCLUSIVE samples, no violation. Sources and installed artifacts stayed unchanged; all processes and five statistics collectors exited. No measured phase completed and no diagnostic latency was imported. Canonical runs retain their 2-second guard.

VMA ring ownership control: a diagnostic-only Java/JNI server flag now permits a controlled ring comparison without changing the default. BENCH_DIAGNOSTIC_VMA_RING_LOGIC=20 on an open-mode bench.sh --only '^java-jni/vma$' run selects the server's per-thread RX ring; clients remain unchanged. Use 0 as the shared-ring control. Recipe, launcher invocation and server diagnostic stamp each independently block canonical publication. The launcher rejects other roles/transports/providers and accepts only 0/10/20/30. The installed VMA README and NVIDIA's 9.8.80 manual identify these as interface/socket/thread/core respectively: old comments incorrectly called 10 per-IP and 30 per-socket. Historical numeric settings and results are unchanged. This is a hypothesis test, not an established fix.

Ring20 control outcome (2026-09-10 00:22:14 UTC): the same-port diagnostic failed before Logon, so per-thread RX rings are not a demonstrated fix for this shard setup. VMA's startup log confirmed mode 20, and statistics showed four receiving rings plus a TX-only ring. The live watcher saw five PASS samples and one INCONCLUSIVE, no violation. Client 1 disconnected before Logon; clients 2–4 failed to start their first initiator. No warmup was released or measurement collected. The short-lived client sockets mostly escaped statistics sampling; that absence is not zero traffic. All processes and collectors exited, with source and installed manifests unchanged. The ring0 default remains in place; neither failed diagnostic replaces a latency result.

Shared-ring debug control (2026-09-10 00:31:30 UTC): the same-port follow-up reproduced the warmup stall with the normal 2-second no-progress guard. Each client sent 4,096 requests, received zero replies and stopped with 4,096 outstanding. All 32 connections were accepted by listener fd 35; correct pins on four workers do not establish distributed shard admission. VMA logged 32 successful per-connection flow registrations, which does not prove subsequent payload delivery. The live watcher recorded five PASS samples, one INCONCLUSIVE and no violation. Source and installed manifests stayed unchanged; all clients, the server and five statistics readers exited. Debug logging makes this a nonpublishing diagnostic; no measured phase or replacement latency exists. The original transport stall remains unresolved.

Native pool failure propagation: a rejected C++ worker CPU request now stops the pool before its application callback. JNI checks the pool runner's failure return; .NET Native stops on a fatal readiness/transport poll error and retains it through Step, error counts and teardown. Stopped or failed pools no longer report connected/active from stale worker snapshots. A disappearing peer's queued-command error is not treated as a pool-wide transport failure. Five focused C++/JNI/native test executables and nine launcher/placement checks passed on housekeeping cores; the invalid-pin regression fails against the previous C++ header. The GCC 14 rebuild and staging verification passed for all 48 installed files, from source 3e4de53c; network remeasurement follows. It improves failure detection, not evidence that the intermittent VMA payload stall is fixed.

.NET Native/kernel POOL remeasurement (2026-09-10 01:02:28 UTC): the fresh full-recipe cell completed all 1.92 million warmup and 1.92 million measured requests, with no shedding, refusal or outstanding replies. Exact raw replay verifies all four service/response ranks; the 36 full one-second windows contain 49,999–50,000 task starts, 49,998–50,001 completions and zero generator backlog at their ends. Service p50/p99/p99.9/max is 33.202 / 46.025 / 53.205 / 150.796 µs; response is 38.126 / 54.407 / 59.480 / 155.903 µs. The row remains partial: 30 sessions pass stability and two warn on p99 only, with no stability failure. Those two p99 values move from 36.317/37.870 µs in run 1 to approximately 28.9/30.3 µs in runs 2–3 (CV 13.67%/13.34%), while their p50 CV stays below 0.2%. This localizes the qualification to run-to-run tail variation, not incomplete delivery; its physical cause is not established. The live audit records 11 PASS, ten INCONCLUSIVE samples and no violation; startup readbacks verify workers 10–13, clients 15–18 and coordinator 19. The server exits with zero errors. Sources and installed artifacts stayed unchanged.

.NET Native/VMA POOL remeasurement (2026-09-10 01:08:35 UTC): the fresh full-recipe cell completed all 1.92 million warmup and 1.92 million measured requests, with zero shedding, refusal or outstanding replies. Exact raw replay verifies all four service/response ranks. Service p50/p99/p99.9/max is 14.957 / 29.142 / 35.548 / 48.963 µs; response is 15.907 / 30.590 / 37.520 / 50.992 µs. The 36 full one-second windows contain 49,999–50,000 task starts and 49,999–50,001 completions, with zero generator backlog and at most one outstanding request at their ends. The row remains partial: 31 sessions pass stability, and client process 1, local session 3 (zero-based), fails p50 only. Its per-run medians are 12.689 / 15.242 / 15.207 µs, giving 10.18% CV against the unchanged 10% limit; its p99 and p99.9 pass. This identifies run-to-run median variation, not missing delivery; its physical cause is not established. The live audit records ten PASS, 13 INCONCLUSIVE samples and no violation; startup readbacks verify workers 10–13, clients 15–18, coordinator 19 and all five VMA maintenance threads on 14. The server accepted and completed 32 sessions and exited with zero errors. The canonical shared-ring configuration and latency-profile receive polling were retained. Sources and installed artifacts stayed unchanged. The earlier payload stall did not recur in this attempt; one completion does not establish that the intermittent failure is fixed.

C++/VMA POOL remeasurement (2026-09-10 01:19:58 UTC): the fresh full-recipe cell completed all 1.92 million warmup and 1.92 million measured requests, with zero shedding, refusal or outstanding replies. Exact raw replay verifies service p50/p99/p99.9/max 14.057 / 37.842 / 48.102 / 66.502 µs and response 14.760 / 38.670 / 49.242 / 67.344 µs. The row remains partial: 27 sessions pass and five warn on p50 only (CV 6.09–7.12%); no session fails stability. All p99/p99.9 stability checks pass. The 36 full one-second windows contain 49,999–50,000 task starts and 49,998–50,001 completions, with zero generator backlog and at most one outstanding request at their ends. Startup readbacks confirm workers 10–13, coordinator 19, clients 15–18 and VMA maintenance on 14; the watcher records 15 PASS, eight INCONCLUSIVE samples and no violation. The canonical ring20 configuration was retained. Sources and installed files stayed unchanged. The earlier coordinator/worker collision did not recur; complete delivery and correct placement do not explain the remaining median variation.

VMA explanation correction: version-matched vendor source review shows that TCP MSG_DONTWAIT receives return an empty-CQ EAGAIN before consulting VMA_RX_POLL. The historical claim that -1 makes every empty native TCP receive spin forever was too broad; blocking accept/connect paths differ. Existing settings are unchanged. The shared-ring unicast dispatcher also visits registered sinks in order, not with a kernel-style reuse-port hash selector. Together with all 32 JNI shard connections landing on one listener, this means four bound listeners and four pinned workers are insufficient proof of distributed shard service. The contract now qualifies that topology explicitly. Neither finding alone establishes the cause of the JNI payload stall.

JNI/VMA POOL role-name failure (2026-09-10 01:29:08 UTC): the next attempt was stopped before import after 17 live audit violations. Coordinator TID 592744 was correctly on core 19 and VMA maintenance TID 592749 on core 14, but both were named hft-pool-coord: the library helper inherited its creator's role name during listener initialization. The source correction preserves the pre-bind affinity check, then assigns/verifies the coordinator name after transport initialization and before publishing readiness or starting workers. The socket-free regression fails against the previous source and passes with TLS disabled/enabled; two placement tests and all 23 unchanged live-pinning tests pass. Rebuild, installation and a fresh live run are still required. The bounded old harness exited; its data did not replace the matrix row. At that point the driver gated startup readbacks, not the separate live watcher, so this rejection required explicit interruption. It is a role-identity fix, not proof that the separate VMA stall is resolved.

JNI role fix — installed and live-verified (2026-09-10 01:43:13 UTC): the GCC 14 rebuild and installation passed with all 48 installed artifacts verified; source and compiler configuration stayed unchanged during the build. The fresh full-recipe attempt had one hft-pool-coord on 19, maintenance named java on 14, workers 10–13 and clients 15–18. The live watcher records seven PASS, two INCONCLUSIVE samples and no violation. The role-name defect is corrected, but the separate payload stall recurred: clients 1 and 3 each sent 4,096 warmup requests with zero replies; clients 2 and 4 drained 131,069 / 131,012 actual sends while shedding 28,931 / 28,988 warmup slots. No measured phase started. All clients emitted zero-sample failure rows, and the parameter audit rejected the attempt before import. The matrix retains its older qualified observation, not a successful rerun. Every process exited. No rate, cap, watchdog, stability threshold or VMA setting was relaxed; no statistics/debug observer was attached. Correct role names and pins do not identify the stage losing payload progress.

Live-audit publication gate (2026-09-10): the publishing driver now owns a fresh watcher for each cell, on verified housekeeping CPUs, and validates its complete log before importing any latency. A detected placement violation, auditor error, malformed log, crashed/stuck watcher or absence of positive samples stops publication and preserves the previous row. An INCONCLUSIVE task observation no longer hides a violation elsewhere in the same sample. Graceful shutdown finishes and retains the last sample, so its verdict cannot be discarded at cell completion. Counts, log hash and observer affinity are retained with each accepted result. Startup/teardown INCONCLUSIVE samples stay visible; this is discrete global FIX-process sampling, not continuous or per-phase coverage, and the separate startup role audit remains mandatory. Inert regression tests exercise rejection before import and final sample retention. The full C++/kernel pool below live-verifies the integrated gate; this does not resolve the separate JNI/VMA payload stall. See the run and evidence instructions.

C++/kernel POOL — integrated live gate (2026-09-10 02:04:02 UTC): the fresh Mellanox-path cell completed all 1.92 million warmup and 1.92 million measured requests, with zero shedding, refusal or final outstanding work. The owned watcher retained 14 PASS samples, no INCONCLUSIVE, invalid or violation samples, and stopped cleanly before publication. Its actual role readbacks show coordinator 19, workers 10–13 and clients 15–18. Source/runtime hashes remained frozen; parameter and startup-role audits passed. Service p50/p99/p99.9/max is 28.285 / 38.942 / 45.985 / 154.829 µs; response is 32.467 / 45.894 / 49.520 / 158.812 µs. All raw ranks and logged means replay exactly. The 36 full one-second windows contain 49,999–50,000 task starts and 49,998–50,000 completions, with no generator backlog at their boundaries. This is fixed-target evidence, not maximum throughput.

Why that cell remains partial: 31 sessions pass stability; process 4 / local session 5 has a p99-only warning (CV 13.41%, unchanged warning threshold 10%). Its three p99 values are 22.281 / 28.356 / 28.454 µs, while p50 is 20.977 / 20.596 / 20.643 µs and p99.9 is 29.454 / 29.104 / 29.790 µs. The raw CDF review finds 191 / 226 / 241 of 20,000 samples above 25 µs: the slower group crosses the 1% rank boundary, moving p99 between two latency groups without a comparable median or p99.9 shift. The 25 µs cut is a post-hoc description, not a new gate. The warning is numerically correct and remains visible. NIC, interrupt and scheduling causes of the slower group are not established by this replay.

JNI/kernel POOL control (2026-09-10 02:13:56 UTC): the current JNI binary on the Mellanox kernel path completed all 1.92 million warmup and 1.92 million measured requests, with no shedding, refusal, stale replies or final outstanding work. This is the same JNI build that stalled during VMA warmup, not a replacement binary. The live gate, startup-role audit and parameter audit pass; raw ranks and logged means replay exactly. Service p50/p99/p99.9/max is 34.423 / 48.906 / 50.672 / 256.980 µs; response is 39.678 / 53.191 / 57.730 / 264.114 µs. Delivery is complete, but the result remains partial: 30 sessions pass stability, process 1 / local session 3 has a p50 FAIL (33.436 / 33.968 / 26.557 µs, CV 13.20%), and process 2 / local session 7 has a p99 WARN (37.347 / 29.835 / 36.538 µs, CV 11.93%). No gate was relaxed. This establishes a working kernel control for the pool's payload path; it does not prove the VMA stall's cause or resolve the kernel variability.

JNI/VMA POOL ring20 diagnostic (2026-09-10 02:20:39 UTC): changing only the server to per-thread receive rings did not produce a valid cell. All clients entered warmup, but client 1 aborted after 16,760 sends / 16,759 replies, with one request still outstanding, when the correlation table wrapped onto its live entry. The guard correctly refused to overwrite that request. The other clients drained 156,835 / 156,969 / 157,047 sends while shedding 3,165 / 3,031 / 2,953 warmup slots; no measured phase began. The live watcher retained seven PASS, two INCONCLUSIVE and no violations. This exposed two failure-evidence defects: the rejected new slot was counted consumed before any send or shed, and the collision log omitted old/new IDs and provider drop/sequence telemetry. Those need correction, not removal of the correlation guard. Source/runtime stayed frozen, canonical audits refused the diagnostic, and no latency was imported. Ring defaults, rate, cap and the two-second watchdog remain unchanged; the missing reply's cause is unproven.

Correlation-failure evidence correction: the Java client now rolls back only the never-sent replacement slot and marker when the owned-slot guard fires, records the existing and attempted IDs plus retained timestamps, and captures provider drop/sequence counters before closing the session. V4 names the failure correlation; it still fails, and the old request and its timestamps remain owned. The legacy open loop receives the same cold accounting/diagnostic correction. All 49 socket-free client tests pass; the two affected tests fail against the previous source. Healthy-path tests still observe no diagnostic getter calls. The normal GCC 14 / JDK 17 build from 3e58a95f passed and installed-runtime verification completed at 02:35:39 UTC: only fixbench.jar changed, with all 51 other inventoried artifacts retained byte-for-byte and root-owned paths checked. The JNI library remains the 80d5e7dd build. This is not yet an explanation or fix for the missing reply.

Instrumented JNI/VMA POOL retry (2026-09-10 02:38:09 UTC): the fresh ring20 diagnostic failed before warmup: clients 1, 2 and 3 disconnected before Logon, and client 4 aborted at the shared barrier. There were no measured samples and the new correlation-owner telemetry was not reached. The live watch retained six PASS, two INCONCLUSIVE and no violations; source and installed hashes stayed frozen. This does not verify the collision logging or explain the earlier missing reply. Per-thread rings have not established reliable startup or delivery. The noncanonical run was rejected by the parameter audit and no latency was imported.

JNI false-EOF correction (source regression, 2026-09-10): the acceptor's readiness-driven path could remove a live session after a would-block or interrupted read even though ReadAvailableOrEof returned eof=false. The initiator's polling path made the same inference from readiness. Both now trust the explicit transport EOF result, retaining no-data reads and still disconnecting on real FIN or read error. Actual-loop regressions pass with TLS disabled and enabled, covering pre-Logon/live acceptors and direct/polling initiators; the previous source fails the live-session retention assertion. This is a demonstrated code defect, not yet proof of what caused the retained VMA failures. The normal production rebuild and installation from 228752bc passed at 02:49:34 UTC: all four JNI library paths have the new verified hash; all 48 other inventoried files are unchanged. Fresh network evidence remains required before attributing a benchmark improvement to it.

JNI EOF-fix ring20 follow-up (2026-09-10 02:51:30 UTC): the same-setting diagnostic with the new native library still failed before warmup: clients 1 and 2 timed out waiting for Logon; clients 3 and 4 aborted at the shared barrier. No disconnect-before-Logon or correlation-owner failure was reported, but this one run does not establish causality and does not fix VMA startup. The six PASS / two INCONCLUSIVE live samples contained no violations, and source/runtime hashes remained unchanged. No measured samples or noncanonical latency were imported. The source EOF regression is fixed; the transport startup/payload investigation remains open.

Canonical JNI/VMA POOL after EOF repair (2026-09-10 02:56:35 UTC): the fresh shared-ring run completed all 1,920,000 warmup and 1,920,000 measured requests with no shedding, refused sends, stale replies or final outstanding work. The raw replay confirms every reported percentile and the full one-second windows: 49,999–50,000 starts and completions, with zero boundary backlog. All publication audits passed; the live watcher retained 16 PASS, one INCONCLUSIVE and no violations. Service p50 / p99 / p99.9 / max is 12.790 / 19.703 / 20.270 / 69.650 µs; response is 13.503 / 20.489 / 22.262 / 82.405 µs. The result remains partial: 28 sessions pass stability, three fail p50 and one warns on p50. Complete delivery in this sitting does not establish repeatable VMA reliability.

Why the remaining JNI/VMA medians fail stability: the retained distributions contain lower and upper groups around 10.5 and 12.7 µs, whose proportions cross the 50th-percentile boundary. For example, process 1 / local session 4 has p50 12.654 / 10.558 / 10.535 µs (CV 10.82%), while p99 is nearly unchanged at 14.494 / 14.509 / 14.522 µs. The share below a descriptive 11.5 µs cutoff is 40.425% / 64.185% / 61.765%. The other p50 failures are process 3 / local sessions 3 and 4 (CV 10.51% and 10.63%); process 4 / local session 3 warns at 8.86%. This explains the rank movement, not its physical cause. The cutoff is post-hoc description only: no acceptance gate or partial label changed.

JNI/kernel POOL control after EOF repair (2026-09-10 03:05:49 UTC): the fresh Mellanox control completed all 1,920,000 warmup and 1,920,000 measured requests, with zero sheds, refusals, stale replies or final outstanding work. The exact raw replay confirms service 38.541 / 47.473 / 51.261 / 393.895 µs and response 43.316 / 53.549 / 61.173 / 408.054 µs (p50 / p99 / p99.9 / max). Full one-second windows contain 49,999–50,000 starts and 49,998–50,001 completions, zero generator backlog and one or two in-flight requests at the boundary. Startup, parameter and live audits passed (12 PASS, two INCONCLUSIVE, no violations). Stability remains partial: 26 sessions pass, five warn on p99, and process 3 / local session 2 fails p50, with 24.528 / 31.165 / 31.162 µs across runs (CV 13.23%). Full delivery verifies this control's function, not repeatable latency or the physical cause of the remaining variation.

Trace-publication gate correction (2026-09-10): the diagnostic audit previously accepted --acceptor-diag and --vma-trace DEBUG. It now rejects explicit trace flags/environment, acceptor receive-trace output and verbose VMA records before qualification or import; ordinary VMA INFO startup output remains allowed. This prevents a routing diagnostic from replacing a canonical latency cell. Shard launch and affinity checks still do not prove every listener handled payload, so a routing probe must retain that evidence separately.

JNI/VMA SHARD routing probe (2026-09-10 03:15:51 UTC): the short traced diagnostic completed all 32,000 warmup and 32,000 measured requests with no shedding, refusals, stale replies or outstanding work. However, all 32 connections were accepted by listener fd 35 despite four declared listeners. Complete delivery here is not evidence of four active shards. The live watch retained five PASS, two INCONCLUSIVE and no violations; source and runtime hashes stayed frozen. The parameter audit rejected this deliberately short, traced recipe; no latency was imported. This confirms concentrated admission in this JNI run, not a result for pure Java or proof of long-run VMA reliability.

Pure-Java/VMA SHARD routing attempt (2026-09-10 03:25:36 UTC): the bounded DEBUG probe was inconclusive: its accept-poll logging filled the 128 MiB transcript limit before any accepts were recorded. Four listeners had announced startup, but all four clients failed to start their first session; there were no measured samples. The live watch had five PASS, one INCONCLUSIVE and no violations, with frozen source/runtime hashes and no remaining benchmark processes. This short traced run was rejected by the canonical audit. It neither proves pure-Java listener distribution nor replaces the existing latency observation.

Java/VMA shard participation gate: the Java server now retains whether each in-process listener observed Logon through its existing control-event drain, and prints one record per listener at shutdown. No per-message trace is needed. Parameter and publication checks reject missing, duplicate or idle-listener evidence for Java/VMA SHARD, even with complete client results. This proves listener activity, not per-shard order counts or equal load distribution; the separate full-delivery checks still apply. Older observations are not certified by this new gate. The normal production build and installation from 976f791b passed at 03:36:05 UTC: only fixbench.jar changed; all 51 other runtime files, including JNI, retained their verified hashes. This changes reporting, not VMA's listener routing.

Pure-Java/VMA SHARD cold-report control (2026-09-10 03:38:25 UTC): the fresh short control completed all 32,000 warmup and 32,000 measured requests with no shedding, refusals, stale replies or outstanding work. Only shard 0 observed Logon; shards 1–3 reported false. The new participation check rejected this topology despite complete client delivery. Five live samples passed, two were inconclusive, none violated placement, and source/runtime hashes stayed frozen. There was no per-message DEBUG trace. This establishes the idle-listener problem in this pure-Java run, not just JNI. The short INFO-marked diagnostic was not imported; the older partial latency observation has no equivalent participation proof and must not be treated as a verified four-active-shard comparison.

JNI/VMA SHARD cold-report control (2026-09-10 03:41:15 UTC): the matching short control also reported Logon only on shard 0; shards 1–3 were idle. Unlike the pure-Java sitting, warmup stalled: 16,385 requests were sent, one completed, 15,615 were shed at the unchanged cap, and 16,384 remained outstanding. All four clients hit the unchanged two-second no-progress watchdog. Measurement never began. The live watch retained five PASS, two INCONCLUSIVE and no violations, with frozen source/runtime hashes and complete process cleanup. Participation, phase-provenance and explicit-INFO diagnostic gates rejected publication. The new cold report works for both providers; VMA delivery is still not reliable. This run does not prove that listener concentration caused the stall, and no latency or rate was imported.

Mixed-stack JNI/VMA isolation controls: the harness now offers a nonpublishing BENCH_DIAGNOSTIC_VMA_SIDE=server|client switch for exact Java/JNI/VMA, OPEN v4, 32-session nos_er diagnostics. The named side uses VMA and the other uses kernel TCP on the same Mellanox path, keeping session/core layout, listener topology, rate, cap and watchdog unchanged. Recipe and transcript markers independently block import. This isolates which side is necessary for a stall; it is not a kernel or VMA latency cell. The hardware controls below do not themselves fix transport delivery.

JNI VMA-server / kernel-client control (2026-09-10 03:55:47 UTC): the first mixed-stack diagnostic failed before warmup. Clients 1 and 2 reached the shared preparation barrier; clients 3 and 4 timed out connecting their first session. Only server shard 0 observed Logon. The live watch retained five PASS, two INCONCLUSIVE and no violations; source/runtime hashes stayed frozen and all processes exited. VMA was absent from the clients' selected launch paths. A client-side VMA instance is therefore not required for this startup failure, but this does not establish the cause of the earlier payload stall. No phase was released and no latency was imported.

JNI kernel-server / VMA-client control (2026-09-10 03:59:48 UTC): the reverse mixed-stack diagnostic completed all 32,000 warmup and 32,000 measured requests, with no shedding, refusals, stale replies or outstanding work. All four server listeners observed Logon. The live watch retained five PASS, two INCONCLUSIVE and no violations; source/runtime hashes stayed frozen and all processes exited. Both controls used one server JVM with four independent listeners on the same Mellanox path. This successful short control focuses the next investigation on the VMA server path, but does not establish a specific defect or long-run reliability: changing sides also changes the number of VMA processes and maintenance threads. The diagnostic was rejected by publication checks; no latency was imported.

Canonical JNI/VMA MONO control (2026-09-10 04:07:35 UTC): the fresh full-recipe run with one server owner completed all 1,920,000 warmup and 1,920,000 measured requests, with no shedding, refusals, stale replies or final outstanding work. The exact raw replay verifies service 10.455 / 16.815 / 17.288 / 65.925 µs and response 11.642 / 18.503 / 19.358 / 85.324 µs (p50 / p99 / p99.9 / max). The 36 full one-second windows contain 49,999–50,000 starts and completions, with zero generator backlog or outstanding work at their boundaries. Parameter, startup and live audits passed (15 PASS, two INCONCLUSIVE, no violations); source/runtime hashes remained frozen. The HTML matrix is updated.

MONO remains partial for latency stability, not missing replies: 28 sessions pass, three fail p50 and one warns on p99. Local session 5 in client processes 2–4 has medians around 8.4–8.7 and 10.5–10.7 µs across runs, while all three sessions' p99/p99.9 stability checks pass. For example, process 2 has p50 8.565 / 8.392 / 10.549 µs (CV 13.07%), versus p99 10.921 / 10.943 / 11.066 µs. These are verified rank changes, not proof of their physical cause. This sitting did not reproduce the multi-listener VMA stall; it neither qualifies SHARD nor establishes repeatable transport reliability. No stability gate or partial label was relaxed.

JNI/VMA SHARD per-connection stall (2026-09-10 04:18:50 UTC): the fresh observer diagnostic retained the existing 60-second diagnostic watchdog and one run. Client C4S8 completed all 20,000 warmup requests; the other 31 sessions completed none. Of 640,000 planned requests, 35,872 were sent, 604,128 were shed and 15,872 remained outstanding. No measured phase started. Server VMA statistics show one socket with 20,001 receives and 31 with only their initial receive; those counts are not independent wire captures. Only listener 0 observed Logon. The live audit retained 13 PASS, two INCONCLUSIVE and no violations, with frozen source/runtime and complete process cleanup.

What that rules out—and what it does not: across a 63.35-second CPU-counter interval, every server worker used essentially one full core. All four epoll miss counters advanced by over 129 million during the statistics capture; one epoll also reported successful readiness. This is not a whole-server blocking hang, though contention and per-connection transport faults remain possible. No CQ-drop or buffer-exhaustion counter was observed. The requested native performance profile was denied by host access controls despite perf_event_paranoid=-1, so there is no stack or lock-owner evidence and the exact denying layer is not established. No security settings were changed. The extended-guard diagnostic was rejected by publication checks; no latency or successful SHARD result was imported.

Explicit listener-address correction (source regression, 2026-09-10): all six tested C++ plain-socket server shapes ignored the requested listen address in the installed binary and bound to 0.0.0.0. The source now forwards the address through single poll/direct-read, legacy multi, mono, pool and shard. The shared C++ / Java/JNI / .NET Native pool also now honors SocketAcceptHost, retaining socket policy and bind-error handling. All 24 loopback-only CLI checks pass against the corrected test build, using actual kernel LISTEN readback; wildcard omission, invalid addresses and overlong-address rejection are covered. Run taskset -c 0-9 make -C bench/cpp/fixbench plain-server-bind-test after building the C++ binary on SCAN. The normal GCC 14 production builds passed; installation was verified at 04:46:54 UTC: 14 native runtime files changed, the other 38 remained byte-for-byte identical, and root-owned paths were checked. All 24 CLI cases also pass against the optimized C++ binary. The first post-fix hardware cells are recorded below. Java's launcher still requests its default wildcard, so this fix alone does not change Java's actual bound endpoint. This is not yet an explanation or fix for the JNI/VMA SHARD stall, and no latency or partial label was changed.

C++/VMA POOL after the bind correction (2026-09-10 04:51:12 UTC): the fresh canonical cell completed all 1,920,000 warmup and 1,920,000 measured requests, with no sheds, send refusals or final outstanding work. The exact raw replay verifies service 14.251 / 37.970 / 48.158 / 69.920 µs and response 15.102 / 38.568 / 49.502 / 70.782 µs (p50 / p99 / p99.9 / max). The live audit has 16 PASS and no violations; workers 10–13, clients 15–18, coordinator 19 and VMA maintenance 14 remain distinct. Source/runtime hashes stayed frozen, with the canonical ring20 setting retained. The HTML is updated.

The remaining C++/VMA POOL partial is median stability: 29 sessions pass and three warn on p50 only (CV 6.19–6.72%); none fails, and every p99/p99.9 stability check passes. The 36 full one-second windows contain 49,999–50,000 starts and 49,998–50,001 completions, no generator backlog and at most one outstanding request at their ends. An unreported C++ stale counter remains unknown, not asserted zero. Complete delivery and correct placement do not explain the median variation; the binding repair has not removed it, and no gate was relaxed.

HP sync and VMA pool policy (2026-09-10): 411f1fe0 withdrew the C++ ring20 default after the reported stall improvement did not reproduce on HP. SCAN pulled that change after the 04:51:12 UTC cell above completed. That cell retains its explicit ring20 provenance; it is not evidence for the restored default. The fresh default-policy sitting is recorded below. Neither setting has established reliable stall recovery; the HP investigation and both hosts' results were preserved when merging the generated handbook.

.NET Native/VMA POOL after the bind correction (2026-09-10 04:58:16 UTC): the fresh canonical cell completed all 1,920,000 warmup and 1,920,000 measured requests without sheds, refusals or final outstanding work. Its exact replay verifies service 15.279 / 29.043 / 34.213 / 54.919 µs and response 16.047 / 30.022 / 36.099 / 55.638 µs (p50 / p99 / p99.9 / max). Parameter/startup audits pass; the live watch has 11 PASS, five INCONCLUSIVE and no violations. Source/runtime stayed frozen. The 36 full one-second windows retain 49,999–50,000 starts, 49,998–50,001 completions, no generator backlog and at most one outstanding request at their ends. The HTML is updated.

.NET Native's partial is also median stability: 30 sessions pass, one warns and one fails p50. The failing session (process 2, local index 1) has medians 12.589 / 12.395 / 14.870 µs (CV 10.3605%), while its p99 values are 27.261 / 27.263 / 27.231 µs. Every session passes p99/p99.9 stability. This confirms the reported rank variation and complete delivery, not its physical cause or repeatable stall recovery. The bind fix has not removed that variation; the partial label and all gates remain unchanged.

C++/VMA POOL restored-default sitting (2026-09-10 05:04:14 UTC): the fresh full-recipe cell uses the merged harness with no explicit ring override, not the withdrawn ring20 default. All 1,920,000 warmup and 1,920,000 measured requests complete, with no sheds, refusals or final outstanding work. The exact replay verifies service 10.856 / 25.097 / 31.475 / 46.605 µs and response 11.698 / 25.819 / 32.371 / 47.743 µs (p50 / p99 / p99.9 / max). The live watch has 15 PASS, one INCONCLUSIVE and no violations. All 36 full one-second windows retain 49,999–50,000 starts, 49,999–50,001 completions, no generator backlog and at most one outstanding request at their ends. Source/runtime hashes stayed frozen; the same staged native build was used as in the preceding ring20 sitting. The HTML now shows this fresh result.

Lower observed latency, still partial: 29 sessions pass and three warn on p50 (CV 5.06–6.39%); none fails and all p99/p99.9 stability checks pass. This sitting is lower than the preceding ring20 observation, consistent with SCAN's earlier diagnostic comparison, but does not establish a reliable stall remedy or a maximum-throughput result. Both sittings remain separately retained; neither the recipe nor the stability gates were relaxed.

Partial-session variation also occurs within runs: equal-sample fifths of the raw captures show median shifts in both the .NET Native cell and the preceding C++ ring20 cell. For example, .NET Native process 2 / local session 1, run 1, has service medians 14.916 / 14.988 / 14.958 / 14.979 / 12.340 µs across its fifths, while scheduler-delay medians stay between 1.529 and 1.597 µs. C++ process 1 / local session 5, run 3, moves 12.894 / 14.272 / 14.291 / 14.340 / 12.914 µs with scheduler medians 1.030–1.074 µs. These chunk quantiles are not averaged to obtain the whole-run quantile. The evidence does not support a universal explanation of one fixed slow state for each entire run. Service-loop phase, NIC and scheduling causes remain hypotheses; these paired timestamps neither establish a physical cause nor alter any partial label.

JNI POOL latency groups versus send lateness: an offline paired review checked 300,000 samples from the sessions with median instability: response minus service equals scheduler delay for every checked sample. In VMA process 1 / local session 4, run 1, the lower and upper service groups average 10.393 and 12.869 µs, while their mean scheduler delays are 1.253 and 1.262 µs. Kernel process 3 / local session 2 similarly has service groups averaging 24.270 and 33.124 µs versus scheduler delays of 4.817 and 4.688 µs. The group separation is therefore not simply different pre-send lateness; these timestamps do not distinguish receive-loop, transport or server cost. The cutoffs are descriptive only and all partial/stability labels remain.

Readiness-control errors no longer look idle (source regression, 2026-09-10): the epoll policy discarded failed registrations/removals and returned zero after a failed epoll creation. A real /dev/null registration reproduced that hidden failure. The source now retains the first control error until explicit reset, clears stale ready bits, and reports a negative result from Poll(). ADD/MOD recovery is bounded to one repair instead of recursive alternation. Successful repairs, absent deletions, interrupted waits and ordinary pipe/eventfd readiness remain covered and passing.

The fault suite passes both unoptimized and normal GCC 14 optimized builds. Actual JNI acceptor/initiator regressions pass with TLS disabled and enabled, verifying terminal error reporting before receive or accept. This closes a diagnostic blind spot, not an established cause of the JNI/VMA SHARD stall. The normal GCC 14 production rebuild passed, with staging verified at 05:33:06 UTC: the four JNI runtime library paths changed, the other 48 artifacts remained byte-identical, and root-owned paths were checked. The first fresh hardware control is recorded below; no latency or partial label was changed by the source tests.

JNI/kernel POOL after epoll error handling (2026-09-10 05:37:01 UTC): the full canonical Mellanox control completed all 1,920,000 warmup and 1,920,000 measured requests, with no sheds, send refusals, stale replies or final outstanding work. The exact raw replay verifies service 45.799 / 50.986 / 54.374 / 123.966 µs and response 50.191 / 56.377 / 60.585 / 154.276 µs (p50 / p99 / p99.9 / max). The live audit retained 12 PASS, two INCONCLUSIVE and no violations; source/runtime hashes stayed frozen. All 36 full one-second windows have 49,999–50,000 starts and 49,997–50,001 completions, no generator backlog, and one or two outstanding requests at their ends. The HTML is updated.

The kernel control remains partial: 29 sessions pass, two fail p50 and one warns on p99. Process 1 / local session 3 has medians 32.063 / 31.592 / 24.678 µs (CV 14.04%), while its p99 values remain 34.576 / 34.556 / 34.504 µs. Process 2 / local session 3 shows the opposite run ordering, 24.058 / 31.112 / 30.892 µs. Complete delivery verifies this control path, not a latency improvement or a fix for the VMA stall. The new headline p50 is higher than the earlier 38.541 µs sitting; these separate unstable sittings are not an isolated causal A/B of the epoll change. The headline p50 belongs to a different, stable session (process 1 / local session 5: 45.404 / 45.918 / 46.076 µs, CV 0.77%). The cell takes the worst session's stability verdict independently of which session sets its headline quantile. No stability threshold was relaxed.

Kernel within-run shifts, not just different run epochs: the paired fifths replay checks every response = service + scheduler sample in the three unstable sessions. Process 1 / local session 3 / run 1 has fifth medians 32.174 / 32.109 / 24.568 / 32.115 / 32.156 µs, while scheduler medians stay between 5.267 and 5.621 µs. Process 2 / local session 3 / run 2 also switches within the run, 24.310 / 31.215 / 31.129 / 24.074 / 31.229 µs, with scheduler medians 4.507–4.650 µs. These observations rule out a universally fixed whole-run slow state, not all phase effects; they do not identify the physical receive-side cost. All warnings remain visible.

JNI/VMA SHARD after epoll error handling (2026-09-10 05:41:33 UTC): the fresh canonical attempt still failed during its first warmup under the normal 2,000 ms watchdog. The failure review retains 16,385 sends, one completion, 100,007 shed slots and 16,384 outstanding requests. Another 523,608 future slots were never consumed; they are not counted as sheds. No measured phase started. All four listeners reported their pins, but only listener 0 observed Logon. The participation and aborted-phase checks correctly rejected publication; no latency was imported.

Independent startup-affinity replay passed; the live watch retained five PASS, two INCONCLUSIVE and no violations. Source/runtime hashes stayed frozen and all benchmark processes exited. The retained server log contains no reported acceptor error event; that is not proof that every internal operation succeeded. This sitting demonstrates that the epoll-control correction has not resolved the VMA stall or the missing listener participation. Failure archive bytes remain explicitly unvalidated phase evidence.

.NET Native/kernel SHARD gap filled (2026-09-10 05:47:43 UTC): the full canonical Mellanox sitting completed all 1,920,000 warmup and 1,920,000 measured requests without sheds, refusals or final outstanding work. Its exact raw replay verifies service 33.236 / 40.177 / 49.193 / 126.020 µs and response 42.323 / 49.382 / 55.165 / 134.208 µs (p50 / p99 / p99.9 / max). All four server shards handled connections (8 / 9 / 8 / 7); each exited with rc=0 and matching accepted/completed connection counts. These are server lifecycle counts, not independent wire-message counters. Parameter and startup audits passed; the live watch retained 11 PASS, three INCONCLUSIVE and no violations. Source/runtime hashes stayed frozen.

This partial is p99 variation, not p50 or delivery: 27 sessions pass and five warn on p99 (CV 11.09–12.30%); no session fails, and every p50/p99.9 stability check passes. For example, process 1 / local session 5 has p99 38.085 / 31.061 / 30.905 µs, but medians 28.652 / 28.695 / 28.717 µs. All 36 full one-second windows contain 49,999–50,000 starts and 49,998–50,001 completions, zero generator backlog and one or two outstanding requests at their ends. An unreported .NET Native stale counter remains unknown, not asserted zero. The v4 matrix now contains 55 measured combinations (23 ok, 32 partial); no warning was removed.

The within-run replay shows that this example's higher p99 persists across all five equal blocks of run 1 (37.818–38.341 µs), versus 30.794–30.935 µs throughout run 3. Scheduler p99 remains around 9.8 µs in both runs, and service medians stay near 28.7 µs. The effect is not confined to the first few measured samples; this descriptive comparison does not identify its transport, server or receive-loop cause and is not a reason to discard run 1.

.NET Native/VMA SHARD admission defect (2026-09-10 06:01:53 UTC): a short nonpublishing control completed all 32,000 warmup and 32,000 measured requests, with no sheds, refusals or final outstanding work. The raw delivery replay verifies the complete four-client surface. However, listener 0 accepted and retired all 32 connections; listeners 1–3 accepted none and reported zero live sessions. Four live affinity samples passed, one was inconclusive and none violated placement; source/runtime stayed frozen and all processes exited. Complete delivery here is not evidence of four active shards.

Native VMA publication gate corrected: the old native shard auditor accepted those idle listeners. It now requires positive admission on every .NET Native/VMA shard, sharing the check between parameter audit and publication. Unequal nonzero counts remain valid; kernel/Onload behavior is unchanged. Seven new regression cases fail against the old code, including every headline verdict and exit-anomaly recovery; all 18 shard-audit tests, 87 importer tests and 11 parameter tests pass after the fix. Replaying the actual control now rejects listeners 1–3. Admission counters do not independently prove Logon or order counts; complete client delivery is still required. This repairs the audit, not VMA routing, and does not certify older observations.

Diagnostic limits are not transport faults: an earlier attempt failed before FIX startup because its wrapper imposed a 128 MiB process-wide file-size cap. Even unpreloaded dotnet --info reproduces the failure: memfd_create("doublemapper") is followed by a 2 TiB sparse-size ftruncate, returning EFBIG and SIGXFSZ. The successful control above restored ordinary runtime file-size limits without changing the benchmark rate, cap or watchdog. Its one-run, 1,000-iteration recipe passes the old parity checker but is independently rejected by the full-v4 publication check. Neither attempt's diagnostic latency was imported; the v4 SHARD gap remains unqualified.

Full .NET Native/VMA SHARD confirmation: the fresh full-recipe attempt completed all 1,920,000 warmup and 1,920,000 measured requests, with no sheds, refusals or final outstanding work. Its raw delivery and admission review again finds accepted/completed connections 32 / 0 / 0 / 0 across the four listeners. The full canonical recipe and independent startup-affinity checks pass; the live audit has 15 PASS, one INCONCLUSIVE and no violations. Source/runtime hashes stayed frozen and all benchmark processes exited.

The corrected admission gate rejects listeners 1–3 despite complete clients. This is no longer merely a short-probe observation: the single-listener behavior persists through the complete workload. The reported session stability also includes five warnings and two failures, but latency ranks were not independently replayed for this disqualified topology. No latency was imported, and the .NET Native/VMA SHARD v4 gap remains unqualified.

C++/Rust-native VMA SHARD audit extension: these flavours share the plain C++ shard server, whose older logs showed four startup lines but no per-listener participation. The server now emits a once-per-owner Logon stamp, outside the measured order loop. The parameter/publication gate requires all four owners to report it; full clients and exit-anomaly recovery cannot override an idle owner. Fourteen negative regression cases fail against the old auditor; all 120 shard, importer and parameter tests pass after the fix. This is an evidence correction, not a claim that C++ admission is concentrated. The GCC 14 production rebuild and staging passed: all five C++ artifacts refreshed, the other 47 runtime files unchanged, root-owned paths verified. All 25 actual-binary loopback cases pass, including no participation at bind and one report per active owner after Logon. Fresh full-recipe controls are recorded below.

Full C++ SHARD controls (2026-09-10): the kernel/VMA sitting completed the unchanged three-run 50,000/s aggregate recipe on both transports. Each completed all 1,920,000 warmup and 1,920,000 measured requests, with zero sheds, refusals and final outstanding work. Source/runtime hashes remained frozen, startup affinity passed and all benchmark processes exited.

The kernel control proves Logon on all four owners: service p50/p99/p99.9/max is 32.505 / 39.106 / 44.426 / 113.050 µs; response is 36.323 / 46.163 / 50.924 / 119.436 µs. The HTML was updated at completion. Live affinity has 13 PASS and one INCONCLUSIVE, with no violations. Exact rank replay confirms 29 session PASS, three p99 WARN and no FAIL; every p50 and p99.9 passes. For example, client 2/local session 6 has nearly identical medians (21.548 / 21.552 / 21.565 µs), but p99 moves 29.409 / 27.588 / 22.971 µs across the three runs (CV 12.45%). This explains the warning, not its physical cause; the row stays partial.

The VMA control records Logon only on owner 3; owners 0–2 are idle despite complete delivery. All 15 live affinity samples pass. The repaired gate correctly rejects this topology, so no new VMA latency is imported. The reported nine stability warnings are separate from the four-shard qualification failure. This confirms that concentrated VMA admission is not limited to the Java/.NET Native servers; it does not establish the JNI payload-stall cause or fix vendor routing.

Current v4 comparison corrected: the older C++/VMA SHARD and Java-pure/VMA SHARD transcripts also lack the required Logon evidence. Their exact previous rows, source reports and transcript hashes are retained, but their numeric fields are withdrawn from the current v4 matrix. Missing old evidence does not prove which owners handled those old runs; new concentration observations are not assigned retrospectively. Historical v3/closed tables remain unchanged. The withdrawal checkpoint therefore retained 53 numeric observations: 23 passing and 30 partial, plus two explicitly failed topology qualifications.

Rust-native readiness check (2026-09-10): SCAN currently has no installed rust_fix, no Rust compiler found in the host path, and its installed fixed artifact helper predates the Rust targets. These cells have not been run or counted as qualified v4 observations; the documented v4 warmup-recording parity work also remains outstanding. The sweep manifest now includes rust_fix when present, so replacing or removing that executable cannot escape the frozen-runtime check. A separate installer typo passed rust_fix as the wire tool's file mode; the repository helper now uses the intended 0755. Unprivileged fixtures reproduce both defects; no host privileged-helper or authorization changes were made.

.NET Pure/kernel SHARD gap filled (2026-09-10 06:57:54 UTC): the full canonical run completed all 32 sessions, with 1,920,000 warmup and 1,920,000 measured requests, zero sheds, refusals or final outstanding work. Service p50/p99/p99.9/max is 53.495 / 79.367 / 94.396 / 265.471 µs; response is 59.588 / 87.251 / 99.309 / 270.522 µs. All 32 session stability checks pass. The phase and lifecycle review confirms successful Logon distribution 2 / 14 / 8 / 8; all four owners participated, though load was uneven. Eleven live affinity samples pass, one is inconclusive, and none violate placement. Source/runtime stayed frozen and all processes exited. The HTML updated at completion; current v4 coverage is 54 numeric observations, 24 passing and 30 partial.

Managed VMA SHARD publication gate: the existing shutdown counters now qualify listener participation before publication. Every declared owner must have a nonzero successful-Logon count, with consistent session totals and no reported failures. Unequal counts remain valid. Seven negative cases reproduce the old auditor's false acceptance; all 124 shard, importer and parameter tests pass after the fix. No binary or order-loop changes are involved. The full VMA attempt below is rejected; kernel success is not VMA qualification.

.NET Pure/VMA SHARD full attempt (2026-09-10 07:08:55 UTC): the canonical sitting completed all 1,920,000 measured requests without shedding, refusal or final outstanding work. However, successful Logon distribution was 32 / 0 / 0 / 0. The corrected gate rejects three idle listeners despite all 32 reported session stability checks passing. No latency was imported. The delivery and lifecycle review also records 66,649 cap-shed requests in the first warmup; 1,853,351 of the 1,920,000 planned warmup requests were sent and completed. Later warmups had no shedding. The physical cause of that warmup deficit is not established. Raw measured delivery and response = service + scheduler are verified; percentile ranks were not independently replayed for the rejected topology. Startup affinity passed; live sampling had 13 PASS, two INCONCLUSIVE and no violations. Source/runtime stayed frozen and all benchmark processes exited.

Why .NET Pure/VMA SHARD is failed: this is a four-listener topology rejection, not missing measured replies or failed session stability. The focused fix review confirms that the harness sets reuse-port and starts four pinned owners; version-matched VMA source is consistent with first-listener admission. No flag-only remedy is qualified. A bench-private VMA routing fix is proposed for separate approval and testing; the system library and failed row remain unchanged. A pool, separate ports or kernel fallback would be a different cell.

Current Mellanox coverage: across the original five flavours, 41 of 48 supported kernel/VMA/SocketXtreme combinations have numeric v4 results. Five are VMA SHARD qualification failures; the two newly enabled C++ SocketXtreme pool/shard routes are still TODO pending their first audited runs. JNI, .NET Native and .NET Pure now have explicit failed status-only rows, linked in the source TSV to their retained attempts; no numeric values were added. Together with the two earlier withdrawals, current overall v4 coverage is 54 numeric observations (24 passing, 30 partial) and five failures. These counts do not qualify the remaining partial observations or the unrun Rust-native cells.

Java JIT diagnostic preparation (2026-09-10): the kernel, Onload and VMA Java launcher arms now accept the opt-in flag --jit-log for either provider and role. It enables fixed compilation-event logging to existing stdout, with nanosecond monotonic/uptime stamps and process/thread IDs; it accepts no output path or arbitrary JVM flags. Normal launches are unchanged. The parameter auditor and importer reject JIT-logged observations, including retained runtime output without an invocation line. They now also explicitly reject the older opt-in GC-logged observations. All 144 launcher, diagnostic, importer and parameter tests pass. This prepares a separate first-run-tail investigation; it is not yet JIT-cause evidence or a new matrix measurement.

Java JIT kernel collection (2026-09-10 07:32:12 UTC): the instrumented single-session control completed all 60,000 warmup and 60,000 measured requests at the unchanged 50,000/s target. No phase shed, refusal or outstanding work was reported; all measured raw triplets reconcile and stale replies are zero. The existing single-session cap is 1,024 (the 32-session/four-client layout uses 512 per session). Startup affinity passed, with three live PASS, two INCONCLUSIVE and no violations. Source/runtime stayed frozen and all benchmark processes exited. The publication gate rejects the logged run as intended; matrix numbers remain unchanged pending event analysis.

Java JIT correlation, kernel/VMA (2026-09-10): the VMA control also finished at 07:34:28 UTC with 60,000 warmup and 60,000 measured replies, no shedding/refusal/outstanding work and zero stale replies. Startup affinity passed; live sampling had six PASS, two INCONCLUSIVE and no violations. The retained raw/event review places each transport's first-run maximum inside log timestamps for the same client methods: dispatchParsed, isApplicationMessage and isPossResendGapReplayApplication. The zero-based reply ordinals are 990 and 991. There are 39/41 application JIT records during first-run measurement, versus zero in runs two and three on both transports. Raw service p99.9 remains unstable: kernel 150.171 / 37.012 / 36.587 µs; VMA 55.693 / 10.135 / 10.471 µs. These are diagnostic values only. Log timestamps describe events, not compilation durations or proof of a causative pause; logging itself can perturb execution. A separate compilation policy control is needed before treating this correlation as a remedy.

The opt-in --jit-c2-only control requires --jit-log and adds only -XX:-TieredCompilation to those diagnostic JVMs. It emits a second noncanonical stamp and cannot be published, even if the logging invocation is absent from retained evidence. It does not increase warmup, discard samples or change the default JVM policy.

Non-tiered kernel control (2026-09-10 07:42:55 UTC): complete warmup/measured delivery and valid raw samples again, but not a stable result: service p99.9 is 57.169 / 37.192 / 34.896 µs, CV 28.43% (WARN). Application JIT records during first-run measurement fall from 39 to three; the first-run maximum moves to zero-based ordinal 63. This one diagnostic pair supports investigating compilation policy, not adopting it as a complete fix. All canonical matrix values and labels remain unchanged.

Non-tiered VMA control (2026-09-10 07:44:55 UTC): all 60,000 warmup and 60,000 measured requests complete without shedding, refusal, outstanding work or stale replies. Service p99.9 is 14.402 / 9.609 / 9.856 µs (CV 23.91%, PASS); p50/p99 also pass. The two policy controls each record only three application JIT events in first-run measurement, versus 39/41 in the logged default-policy controls. Startup and sampled live affinity pass; source/runtime stayed frozen and all processes exited. This is an encouraging diagnostic comparison, not repeatability proof or a canonical result. The JVM default and matrix rows remain unchanged.

Remaining early spike: a newly exercised cache path. The client intentionally resets order IDs at each phase, while the server's bounded duplicate-order cache survives. A replay against the installed Java class finds zero duplicate hits in the initial 20,000 unique-ID warmup and 174 in the following measured sequence; the first retained ID is order 61. On both non-tiered controls, order 61's raw service interval contains the SeenNewOrderSingleClOrdIds::remember made-not-entrant event, followed by recompilation. That order's service time is 115.536 µs on kernel and 91.852 µs on VMA; later overlapping requests contain the run maxima. This narrows a warmup branch-profile gap, but the exact JVM trap reason/bytecode is not captured. The reflection probe validates cache state only, not timing. No duplicate-suppression semantics, marker resets, warmup counts or stability gates were changed to hide the transition.

Cache trap identified in a CPU-only replay (2026-09-10): compilation XML from the installed class records unstable_if at bytecode 89, the cached-hash equality check, when reused IDs arrive. An earlier warmup trap at bytecode 119 covers the empty-slot-to-eviction transition. This matches the OpenJDK branch parser's treatment of previously untaken paths, but it is not an exact trap capture from the earlier network runs. A separate CPU control with PerMethodTrapLimit=0 records no runtime cache traps. The fixture uses MethodHandles and a verified no-inlining directive; its timings are not network measurements. The initial invalid directive and corrected XML-reader mistake are retained with the evidence.

The fixed opt-in flag --jit-trap-limit-zero requires --jit-log, adds only -XX:PerMethodTrapLimit=0 and emits an independently rejected noncanonical policy stamp. It accepts no arbitrary JVM option. This prepares the full network comparison; it does not change the default policy, protocol checks, marker resets or canonical warmup.

Full kernel speculation control (2026-09-10 08:14:36 UTC): logged non-tiered compilation plus PerMethodTrapLimit=0 completes all 60,000 warmup and 60,000 measured requests, with no shedding, refusal, outstanding work or stale replies. Raw service p99.9 is 38.376 / 38.026 / 37.199 µs (CV 1.60%, PASS); p50 and p99 also pass. No application compilation/invalidation event is logged inside any measured interval, and no cache made-not-entrant record occurs. Startup affinity passes; live sampling has three PASS, two INCONCLUSIVE and no violations. Source/runtime stay frozen and cleanup completes. The earlier non-tiered-only control warned at p99.9 CV 28.43%. This remains a separate diagnostic comparison, not an imported value or justification for silently changing canonical JVM policy.

Full VMA speculation control (2026-09-10 08:18:03 UTC): the matching logged non-tiered, zero-trap-limit control also completes all 60,000 warmup and 60,000 measured requests without shedding, refusal, outstanding work or stale replies. Service p99.9 is 10.674 / 10.393 / 10.731 µs (CV 1.71%, PASS); p50/p99 also pass. No application JIT event occurs inside measurement and no cache invalidation is logged. Startup affinity passes; live sampling has six PASS, two INCONCLUSIVE and no violations. Source/runtime remain frozen and cleanup completes. The paired diagnostic report and launcher-only staging proof retain provenance. All run maxima on both transports are the first measured reply, so the residual phase-start spike is not explained by a measured JIT event. A separate comparison retains default tiering; canonical JVM policy and matrix values remain unchanged.

Default-tiered kernel speculation control (2026-09-10 08:22:26 UTC): retaining tiering with the zero-trap-limit diagnostic gives service p99.9 54.734 / 39.815 / 40.216 µs (CV 18.92%, PASS), with p50/p99 passing too. All 60,000 warmup and 60,000 measured requests complete without shedding, refusal, outstanding work or stale replies. Four application JIT events remain inside first-run measurement, none in runs two/three; the maxima are again each run's first reply. Startup affinity passes, with three live PASS, two INCONCLUSIVE and no violations; source/runtime stay frozen and cleanup completes. This tests whether disabling tiering is necessary, not a new canonical policy or imported result.

Default-tiered VMA speculation control (2026-09-10 08:24:30 UTC): the matching run still warns: service p99.9 is 22.267 / 10.462 / 10.863 µs (CV 46.13%, WARN), although p50/p99 pass. All 60,000 warmup and 60,000 measured requests complete with zero shedding, refusal, outstanding work and stale replies. Eleven application JIT events occur in first-run measurement, none in runs two/three. Every run maximum remains its first reply, not overlapping those JIT events. Startup and sampled live affinity pass with no violations; source/runtime are unchanged and all FIX processes exit. The default-tiered paired report shows why the zero-trap-limit flag alone is not a demonstrated stability fix. The combined non-tiered control is a candidate for further qualification, not an adopted default. These diagnostics do not fill or relabel matrix cells.

JVM policy publication guard: a retained bare -XX:-TieredCompilation argument is now rejected even without launcher stamps or JIT output, matching the existing zero-trap-limit guard. Regression fixtures reproduced the prior acceptance in both successful and failed-result imports. They now require the matrix to remain untouched.

Phase-start qualification: the v4 coordinator's current two-second future-epoch lead leaves a gap without scheduled application sends after warmup. The Java client can poll receives during that gap, but per-run warmup alone does not prove its first application send is hot. This is a source-level reason to investigate phase-start timing separately, not proof of the observed spike's cause. First replies and the shared release policy remain unchanged.

Opt-in first-reply timing probe: --diagnostic-open-phase-timing enables preallocated client-side timeline storage only for single-session OPEN v4 nos_er. Warmup and measurement use identical stores; output happens after each phase and includes its first 64 replies, fixed later samples and maximum. The four intervals cover actual-offer-to-send-return, the wait until the reply's receive step, that receive call, and polling/correlation/extraction through completion. They do not isolate network or server time, and extra clock reads can perturb latency. The full raw series remains unchanged; all trace evidence is rejected from publication independently of JIT logging. The default allocates no trace storage or extra clock reads. Engine instrumentation is not enabled.

Initial phase-trace collection (2026-09-10 08:55:51 UTC): the kernel process completed, but post-collection validation rejected the probe's initial bench_open_phase_timing prefix as a malformed v4 control record. The attempt and its staged-artifact proof remain retained, not imported or relabelled. Diagnostic records now use bench_java_phase_timing; a regression feeds the actual launcher stamp to the strict schedule auditor and separately requires publication rejection. The control grammar was not relaxed. A corrected probe will use a separate source revision and evidence directory.

Corrected kernel phase trace (2026-09-10 09:05:00 UTC): the strict v4 audit and raw timeline replay now pass. All 60,000 warmup and 60,000 measured requests complete with no shedding, refusal, outstanding work or stale replies. Service p99.9 is 39.676 / 38.533 / 38.090 µs (CV 2.11%, PASS), with p50/p99 passing too. No application JIT events occur during measurement. Each run's maximum is still its first reply, with costs spread across several client-side boundaries, not just a wait for the server. The verified trace retains all selected sample checks; the staging proof records the only two replaced artifacts. Startup affinity passes; live sampling has three PASS, two INCONCLUSIVE and no violations. Source/runtime remain frozen and cleanup completes. These extra-clock diagnostics are not matrix values.

Corrected VMA phase trace (2026-09-10 09:08:07 UTC): all 60,000 warmup and 60,000 measured requests complete without shedding, refusal, outstanding work or stale replies. Service p99.9 is 10.725 / 10.879 / 10.887 µs (CV 0.84%, PASS); p50/p99 also pass. All selected measured timelines match the original raw timestamps exactly. No application JIT event occurs during measurement. Every maximum is again the first reply: 56.636 / 57.471 / 57.064 µs. Startup affinity passes; live sampling has six PASS, two INCONCLUSIVE and no violations. Source/runtime stay frozen and cleanup completes. See the VMA trace and 99-test JAR build review. The final verification checks evidence hashes, archived test reports and the exact table values below. The first-reply cost spans send and receive processing on both transports; its exact cold-path cause remains open. These instrumented values do not replace canonical matrix rows or justify dropping the first reply.

Java multi-session readiness correction (2026-09-10): C++ already switches from direct reads to a batched readiness snapshot above four peers; pure Java was still trying every socket each sweep. The candidate Java correction keeps direct reads for at most four plaintext peers per owner and uses one nonblocking selectNow() snapshot above that threshold. Pool ownership stays fixed; TLS and blocking/stream-write modes retain their existing paths. It also restores the missing multiplexed read/parse diagnostics and retires a peer that fills its parser buffer with an incomplete header. Both bugs reproduce on the installed pre-change JAR. The Java-only build review records 234 passing engine tests (including eight new readiness/lifecycle tests) and 99 passing bench tests, with no skips. No native library, pacing recipe, JVM policy or stability gate changed. The staging verification confirms that only the Java engine JAR changed; the other 51 installed artifacts are unchanged. The subsequent normal, uninstrumented kernel/VMA sitting finished at 10:38:47 UTC. Seven cells updated the matrix after completion; VMA SHARD was rejected because only listener 0 observed Logon. The raw qualification review verifies every measured sample and exact logged quantile for the seven published cells, including their existing stability classifications. None is saturated. This is not a qualified general latency/stability fix: MONO medians barely change, kernel MONO's lower p99 comes with a worse max, and VMA POOL's p99.9 and stability worsen. Historical rows are retained in the review, but are not a same-sitting A/B control. No samples or warnings were removed.

All seven published cells have complete measured delivery, zero measured shedding, zero stale replies and no raw drops/clamps. VMA MONO sheds 53,644 requests during the first warmup; the other six warmups are complete. The separately rejected VMA SHARD attempt also has complete warmup/measured raw delivery, but three idle listeners; its percentile ranks were not independently replayed and no numeric row was imported. Startup and sampled live placement passed for published cells; the rejected cell's live watch also passed. Source and runtime stayed frozen, and all benchmark processes exited. HP's later c3467059 Java busy-poll change is not in these measured binaries. The publication verification checks 273 retained evidence hashes and exact table values; all 157 offline publication and audit tests pass.

Java busy-poll verification correction (2026-09-10, build validated): HP's c3467059 adds socket setup through the JDK bridge, but silently skips an unavailable bridge and logs only the first socket attempt. SCAN reproduced both gaps with unconnected sockets. The local correction reports each socket's requested and effective SO_BUSY_POLL value, role and outcome. Explicit 0 clears and reads back the option; an unset request adds no socket work. The parameter auditor and importer require complete, matching client/server readbacks for an explicit pure-Java request. Missing, failed, duplicate or unbound evidence is rejected; 135 offline audit tests pass. Real-JDK probes also verify explicit zero, unavailable module access and later socket failures. The Java-only build qualification records 242 passing engine tests with no skips and passing benchmark compatibility tests. Source and installed artifacts stayed unchanged throughout validation; the staging readback then verifies that only the engine JAR changed, with all 51 other runtime artifacts unchanged. Both full-recipe Mellanox SINGLE explicit-zero/50 µs controls completed. Every client/server socket read back its requested value, each control delivered all 60,000 measured requests, and raw replay matches all logged quantiles and stability classifications. No measured shedding, stale replies, drops or clamps occurred; startup and sampled live affinity passed. Neither control is imported into the matrix, and no canonical default changes. The lower observed service median at 50 µs comes with higher tails and a slightly higher response median (34.292 vs 33.830 µs). Both fail stability because p99.9's sample CV is above 50%. These two ordered sittings are not a randomized/repeated comparison or a general tuning fix.

Existing observations keep their original provenance: historical setter-only logs are not proof of the new readback requirement and cannot qualify a fresh import. SCAN's machine file still has no default busy-poll window; HP's single-cell default is 50 µs. This change does not silently copy that tuning to SCAN.

Full32 busy-poll readback check (2026-09-10 11:31:26 UTC): the already-started kernel MONO control finished with 32 server socket records and eight records from each of four clients: all 64 actual values match explicit zero. The retained control completes all 1,920,000 measured requests without shedding, stale replies, raw drops or clamps. Startup and sampled live affinity pass, source/runtime stay frozen, and all processes exit. Its partial latency result remains a separate control, not a matrix import. Work now prioritizes genuinely passing SINGLE cells, including repeat-run stability; no further multi-session sweep is queued.

SINGLE stability qualification: the fixed --jit-latency-control prepares an unlogged repeatability check of non-tiered compilation with PerMethodTrapLimit=0. The earlier logged controls identified compilation/cache-branch effects, but are not canonical results. The new control enables no JIT/GC/phase logging and accepts no arbitrary JVM flags. It remains explicitly noncanonical until qualification and a separate policy decision; all raw samples and existing stability gates remain mandatory.

Unlogged SINGLE controls: three kernel and three VMA sittings pass stability and complete all warmup/measured requests. The raw qualification reproduces every quantile and stability verdict; actual process command lines confirm the fixed policy and absence of JVM logging. The launcher-only stage leaves all 51 other artifacts unchanged. All six independently retained sittings pass; there are no discarded attempts in this qualification. Each completes 60,000 warmup and 60,000 measured requests. This supports an explicit policy adoption followed by fresh canonical runs; the candidate rows themselves remain separate. The trade-off is visible: stable kernel tails come with higher median latency than the default-tiered controls, not a universal latency reduction.

Canonical Java-pure SINGLE policy (2026-09-10): following the six passing unlogged controls, SCAN explicitly selects JAVA_PURE_SINGLE_JVM_POLICY=single-latency-v1. This fixes non-tiered compilation and PerMethodTrapLimit=0 for both SINGLE endpoints, without JIT/GC/phase logging. Both JVMs must report their effective values, bound by the publication audit to the machine fingerprint, recipe, invocation and cell identity. Other flavours, all multi-session shapes and HP retain their defaults. The unchanged 50k/s target, 20k warmup, 20k measured requests per run and three-run stability gates still apply. The audited stage changes only the launcher, bench JAR and SCAN machine file; 49 other artifacts remain unchanged. The passing control rows above are not relabeled as matrix results. Onload remains pending DAC confirmation.

Fresh canonical SINGLE results: kernel completes at 2026-09-10 12:12:13 UTC with status ok, all 60,000 measured requests delivered and stability PASS. Both effective JVM records, startup and sampled live affinity, raw delivery and exact four-quantile replay pass; source/runtime remain frozen and all processes exit. See the kernel evidence. VMA also completes at 2026-09-10 12:14:09 UTC with ok, stability PASS, complete warmup/measured delivery and the same successful audits: VMA evidence. Each result updates the matrix and this handbook immediately. These are fresh observations, separate from all six qualification controls.

.NET Native SINGLE refresh (2026-09-10): the current managed driver passes 213 multi-client and 90 schedule tests. The managed-only stage preserves the former staging tree and leaves every installed native library unchanged. Kernel's fresh full-recipe run completes at 12:20:42 UTC with ok / stability PASS, complete warmup/measured delivery, successful startup/live affinity audits and exact raw quantile replay. Service p50/p99/p99.9/max are 21.575 / 29.878 / 37.142 / 85.196 µs: kernel evidence. This supersedes the prior partial observation, which remains in its original sitting. It does not by itself identify the cause of that older median drift. SocketXtreme completes at 12:22:25 UTC with full delivery but remains partial / stability WARN: service 8.432 / 12.608 / 19.592 / 39.701 µs, retained evidence. The warning is p99 sample CV 17.06% (13.738, 13.959, 10.128 µs per run); p50 CV is 0.17% and p99.9 CV 3.46%. Large delays often recur at 256-request intervals while scheduler delay stays low. This is a batching-path lead, not yet a causal finding. No samples or stability thresholds are changed.

SocketXtreme 64-buffer refill control (2026-09-10 12:34:48 UTC): the fixed, explicitly noncanonical launcher control changes only VMA's RX pool refill batch from its 256 default to 64. Both endpoint startup records confirm 64. It completes every warmup/measured request and passes pinning/raw audits, with service 8.358 / 9.891 / 15.740 / 38.867 µs. p99 CV falls to 2.06%, but p99.9 CV is 29.74%: WARN, not a qualified fix. Retained control; no canonical row or default changes. The matching explicit-256 control also completes delivery and remains WARN: 8.286 / 12.467 / 17.664 / 41.673 µs, p99 CV 16.92%. Its per-run p99 values (13.730, 13.640, 10.032 µs) reproduce the canonical default's high/high/low pattern. The smaller RX refill addresses p99 variability in this comparison but does not qualify overall stability.

SocketXtreme TX-batch control: at fixed RX refill 64, TX completion batching 4 is confirmed by both VMA startup readbacks. All requests complete, but service 8.379 / 9.753 / 15.812 / 41.036 µs still warns on p99.9 CV 25.92%: retained control. This does not qualify a TX tuning fix, and defaults remain unchanged. Source review also finds a per-request id.ToString(...) allocation in the .NET Native sender. Its causal relationship to the periodic stalls remains to be tested; removing it must preserve in-measurement ID formatting.

.NET Native numeric-ID correction: the sender now formats the decimal NOS correlation ID into a stack buffer during the measured send, instead of allocating a managed string for each offer. It calls the unchanged native composition entry point; no native library or VMA default is changed. Seventeen focused tests pass with no skips, including real kernel FIX replies, signed decimal boundary values, unchanged schedule/error handling, and zero managed allocation over 120,000 numeric native calls. The 213 multi-client and 90 schedule regressions also pass. The managed-only stage is verified. The first fresh canonical SocketXtreme SINGLE sitting finishes at 13:05:36 UTC with ok / stability PASS: service 8.741 / 10.446 / 16.575 / 39.672 µs, complete warmup/measured delivery, successful startup/live affinity audits and exact raw replay. Retained evidence. The kernel regression finishes at 13:07:22 UTC with ok / stability PASS, service 21.525 / 28.678 / 31.932 / 67.393 µs, complete delivery and passing affinity/raw audits. The unchanged second SocketXtreme sitting finishes at 13:08:58 UTC: ok / stability PASS, service 8.272 / 10.187 / 14.662 / 37.247 µs, full delivery and successful affinity/raw audits. The third sitting finishes at 13:10:31 UTC with ok / stability PASS, service 8.241 / 9.841 / 16.147 / 35.946 µs, again with full delivery and passing affinity/raw audits. All three consecutive sittings pass: nine measured runs, 180,000 measured replies and 180,000 warmup replies, no shedding or deficits. The repeatability review revalidates every retained evidence hash and reports each run's quantiles and sample CV. VMA settings, native libraries, workload and stability gates remain unchanged. This supports the correction's repeatability, but does not establish that GC caused the former 256-request periodic stalls.

Current Single status (OPEN v4, 2026-09-10 13:53 UTC): 20 of the original 21 supported Single combinations are ok, with no failed Single cells. Java-pure/Onload now passes after the corrected JVM policy is verified on the restored Solarflare link. C++/SocketXtreme is the remaining partial observation: full delivery, but p99.9 variability above the unchanged warning threshold (details below). Historical matrices and unsuccessful diagnostic controls retain their original verdicts; new Rust capability cells are separate coverage.

.NET Native/VMA Single regression (13:16:31 UTC): the same staged numeric-ID driver also passes its full VMA Single cell: 8.467 / 9.173 / 9.661 / 38.092 µs service, ok / stability PASS. All 60,000 warmup and 60,000 measured requests complete; startup/live affinity and raw delivery/quantile audits pass. The native libraries and VMA defaults remain unchanged. See the retained VMA evidence. This completes the new sender's kernel/VMA/SocketXtreme Single regression; Solarflare tests remain paused pending DAC confirmation.

Native SocketXtreme MONO failure (13:19:03 UTC): the 32-session rerun logs on all clients but aborts in the first warmup; three client processes receive no payload replies. No measured samples exist. The failure report retains all phase counts, the prior numeric row and the successful startup/live pinning audits. The matrix now shows a status-only failure, not old latencies presented as a new measurement. The investigation found a separate defect in the direct runner's cached-readiness loop: an always-ready snapshot can survive empty reads and prevent further listener/shared-ring pump calls. A socket-free regression reproduces that starvation before the fix. Empty-read refresh and bounded listener checks pass the regression, EOF checks, fake TCPDirect lifecycle, VMA ownership/poll failure, pool, real-loopback pollless and failure-caller tests. Attribution correction: .NET Native uses hftNetStepAcceptorImpl, not that cached-readiness loop; JNI likewise has its own serve iteration. Both poll readiness on each pass. Therefore the direct-runner correction does not establish the cause of this .NET Native failure. The production rebuild confirms that libhftnet.so is byte-for-byte unchanged. The Mono failure remains unresolved; Single results are unaffected by it.

Post-rebuild Single checks (2026-09-10): the verified production stage replaces five C++ binaries and four JNI library copies; all 43 other runtime files, including .NET Native, remain identical. No vendor libraries, compiler settings or benchmark gates change. The C++/kernel Single regression passes at 13:42:22 UTC with service 19.426 / 27.179 / 29.882 / 56.987 µs, complete measured delivery and passing affinity, raw replay and stability checks. The C++/VMA Single regression also passes at 13:43:57 UTC: service 6.777 / 7.705 / 7.865 / 25.857 µs, with complete measured delivery and the same successful audits. The C++/SocketXtreme Single regression finishes at 13:45:23 UTC with partial / stability WARN, service 5.904 / 7.411 / 13.567 / 28.712 µs. Every warmup and measured request completes, and affinity/raw audits pass. Its three service p99.9 values are 11.564, 11.578 and 17.560 µs: sample CV 25.486% exceeds the existing 25% warning gate, while p50/p99 CVs pass. In run 3, 60 of 62 samples at or above 15 µs cluster on two modulo-256 offsets; scheduler delay for those tail samples stays below 0.554 µs. This resembles the earlier .NET Native SocketXtreme periodicity but does not identify its cause. All samples and the new partial verdict remain published. The JNI/kernel Single regression passes at 13:50:57 UTC: service 25.569 / 31.562 / 37.573 / 104.890 µs, complete measured delivery and successful affinity/raw/stability audits. JNI VMA/SocketXtreme regressions remain pending. These are regression measurements, not evidence that the separate native Mono failure is repaired.

Java-pure/Onload Single repaired (13:53:43 UTC): both Solarflare ports report live carrier, and the physical-port TX counter advances during the link check. The full canonical cell with the qualified single-latency-v1 JVM policy is ok / stability PASS: service 9.967 / 10.736 / 11.235 / 61.055 µs. All 60,000 warmup and 60,000 measured requests complete without shedding; both JVMs read back the required compilation settings, and startup/live affinity and exact raw replay pass. No samples or stability gates change. See the retained Onload evidence.

C++/SocketXtreme Single TX recycling investigation: version-matched VMA source has a 2048-entry TX queue and 256-buffer ring-pool refills; its SocketXtreme RX poll does not drain TX completions. The new fixed --diagnostic-sx-tx-wre 128|2048 C++ launcher control tests whether earlier TX recycling avoids the periodic tail. It changes no vendor binary or canonical setting. Both choices are noncanonical, and invocation, recipe, environment, diagnostic marker and effective nondefault queue readback independently prevent publication. The first 128-entry control completes all warmup/measured requests and passes affinity/raw audits. Its service is 5.929 / 7.390 / 16.414 / 26.204 µs, with stability PASS, but the modulo-256 tail remains in every run. The stable warning status alone is not a repair: this control is not adopted, and the canonical partial stays in the matrix. The explicit 2048-entry default-depth control also passes, with service 5.930 / 7.470 / 11.715 / 24.828 µs and complete delivery plus successful audits. It has lower p99.9 than the smaller queue. Neither diagnostic replaces the canonical row; queue depth is not an established remedy for the observed variability.

The next C++/SocketXtreme control uses the separate receive-counter binary to record bounded send/receive phase timing and thread page-fault deltas. Storage is prepared before readiness; output follows phase completion, and every normal raw sample remains retained. Additional clocks can perturb this diagnostic, so both its tool name and timeline records reject canonical publication. Normal binaries compile out the new probe. This investigation has not yet established the cause or repaired the canonical partial. The completed client trace (14:28:51 UTC) retains all 60,000 warmup and 60,000 measured replies. Exact raw replay gives service 5.958 / 7.511 / 11.533 / 27.872 µs; all 496 selected measured timelines match their raw samples, with no trace omissions. Each measured run has 79 slow positive receive steps, with median duration about 6.5–6.6 µs, plus a separate periodic waiting-time tail. Sends are generally short, and measured client minor/major page-fault deltas are zero. This narrows the investigation but does not separate peer work from network delay. Live affinity passes; the ordinary offline identity audit correctly rejects the counter tool. Explicit read-only diagnostic identity adapters verify startup placement and raw metrics without changing input logs or permitting canonical publication. The original rejection remains retained. The first probe build also exposed a C++11 compatibility error; it was corrected and regression-tested before the successful GCC 14 build and diagnostic-only staging. No normal or vendor library changed.

C++/SocketXtreme delay localized to the vendor polling call (14:44:09 UTC): the second diagnostic and staging proof separates SocketXtreme polling, held-packet release and remaining receive-step work. In each run's recurring slow positive-receive group, median poll time is 5.913–6.120 µs, packet release is 0.107 µs, and remaining receive work is 0.736–0.739 µs. All 492 selected measured timelines match the full raw series; all warmup/measured requests complete, with zero measured client page faults and successful live and explicit diagnostic startup affinity checks. Instrumented service is 6.155 / 7.805 / 11.873 / 29.579 µs. These clocks identify the API interval, not yet the internal VMA operation or possible interruption. A separate periodic waiting-time tail remains. The counter binary alone changed during staging; the other 51 runtime artifacts, including the normal server/client and vendor libraries, remain identical. The canonical row stays partial. Further private VMA instrumentation requires approval; the installed system library has not been modified.

C++/SocketXtreme no-batching control (15:07:53 UTC): disabling buffer batching at both endpoints completes all 60,000 warmup and 60,000 measured requests with startup/live affinity and stability PASS. Service is 6.040 / 7.576 / 15.234 / 25.340 µs, response 6.324 / 7.889 / 15.541 / 26.057 µs. The recurring tail remains: each measured run has 106–158 samples at or above 10 µs, concentrated at a few positions modulo 256. See the full no-batching evidence and staging proof. Only the allowlisted root launcher changed; benchmark and vendor libraries did not. Both independent publication gates reject the diagnostic. The canonical partial row remains visible.

C++/SocketXtreme default-buffer comparison: the matching default-policy control also completes every warmup/measured request with stability and startup/live affinity PASS. Service is 5.932 / 7.575 / 11.993 / 26.616 µs, response 6.234 / 7.905 / 12.304 / 27.170 µs. The default has lower p99.9 than the no-batching candidate, and the periodic tail persists with either policy. Disabling batching is rejected as a remedy. Both controls remain diagnostic, including the successful default run; neither replaces the canonical partial observation. Private instrumentation inside VMA still awaits approval. No library, sample count or stability threshold changed.

Native SocketXtreme slot-reuse correction (2026-09-10): the native transport move leaked a registered socket allocation, and resetting the vacated slot made the next accept allocate a socket absent from the listener's peer registry. The fix preserves the registered object set through both compaction and reset. An exact pre-fix implementation control fails; corrected tests at capacities 32 and 128 deliver to all 32 clients before and after removal/reacceptance, including owner-thread handoff. Partial-borrow moves and exact-once buffer release pass too. AddressSanitizer and leak checking pass on the host; the sandbox's earlier tracing-related LeakSanitizer refusal is retained. See the ownership regression evidence. These tests seed post-Logon state and use an inert VMA API: they do not prove that the earlier pre-disconnect MONO stall is fixed. The GCC 14 production rebuild and staging completed: only the five native-library copies changed; all 47 other installed artifacts were retained.

Native/kernel SINGLE regression (2026-09-10, 15:33 UTC): all 60,000 warmup and 60,000 measured requests complete, with zero shedding or raw-sample loss, at 49,997/s against the 50,000/s target. Startup and sampled live affinity pass. Service p50/p99/p99.9/max are 23.093/29.084/32.306/67.241 µs; response values are 25.092/33.027/35.137/68.524 µs. The cell remains qualified as partial: service medians of 24.862/22.050/22.368 µs produce 6.67% CV, above the unchanged 5% warning threshold; p99 and p99.9 CV are only 0.60% and 0.21%. This is complete delivery with a median-repeatability warning, not a missing sample or throughput failure. See the raw-derived regression and staging evidence. A raw distribution review finds concentrations near 21 and 26 µs: the share below 23 µs changes from 41.27% to 54.73% and 52.01%, moving the median between them. Mean-latency CV is 1.52%, versus p50's 6.67%. This describes the warning's sensitivity, not the cause of the two concentrations; the mean is not substituted for p50. VMA/SocketXtreme hardware regressions remain separate; no vendor library or stability threshold changed.

Native VMA and SocketXtreme SINGLE regressions (2026-09-10): both pass with the rebuilt native library, unchanged runtime settings and stability thresholds. Each completes all 60,000 warmup and 60,000 measured requests without shedding, refusal or raw-sample loss; startup and sampled live affinity pass. The 15:43 UTC VMA result has service p50/p99/p99.9/max 8.353/8.796/9.194/36.831 µs; response values are 8.660/9.184/9.534/38.138 µs. The 15:44 UTC SocketXtreme result has service 8.511/10.589/18.160/37.400 µs and response 8.775/10.871/18.455/39.545 µs. Both achieve 49,999/s against 50,000/s. These Single controls do not establish that the earlier 32-session SocketXtreme stall is fixed.

Native/SocketXtreme MONO full retry (2026-09-10, 15:48 UTC): the rebuilt native runtime completes all 32 sessions, including all 1,920,000 warmup and 1,920,000 measured requests, with no shedding, refusal or raw loss. The canonical v4 cell moves from failed to partial. Service p50/p99/p99.9/max are 12.382/18.308/21.526/39.534 µs; response values are 13.476/19.703/23.212/41.872 µs. Each session targets exactly 1,562.5/s (50,000/s aggregate), with 1,562/s reported after integer formatting. Startup and sampled live affinity pass; raw quantiles independently replay. Stability is 17 PASS / 15 WARN / 0 FAIL: all warnings are p99 CV above 10%, with the largest 18.84%, not missing replies or failed pacing. The previous pre-disconnect stall did not recur, but this run alone does not isolate its cause from the native rebuild. No failed historical evidence is discarded. See the complete regression evidence.

C++/TCPDirect SINGLE confirmation (2026-09-10, 15:41 UTC): the unchanged canonical v4 run passes again: service p50/p99/p99.9/max are 4.771/5.024/5.151/15.886 µs, response values are 5.088/5.401/5.498/16.837 µs, and throughput is 49,999/s against 50,000/s. All 60,000 measured replies are present, raw quantiles independently replay, and startup/sampled live affinity pass. Service CV is 0.30%/0.38%/2.29%: all unchanged stability gates pass. The current v4 matrix is ok; the historical v3 partial observation belongs to a different schedule and is not overwritten. See the confirmation evidence.

C++/Onload POOL backfill (2026-09-10, 15:53 UTC): the previously unmeasured v4 cell passes with all 32 sessions stable and all 1,920,000 warmup plus 1,920,000 measured requests delivered. No shedding, refusal or raw loss occurs. Service p50/p99/p99.9/max are 8.847/10.137/10.485/26.181 µs; response values are 9.646/11.300/11.545/27.817 µs. The target is 1,562.5/session/s, with 1,562/s reported after integer formatting. The coordinator reads back core 19 and workers 10/11/12/13; startup and sampled live pinning pass. Raw-derived quantiles and all 32 service stability verdicts independently agree with the logs. See the full backfill evidence.

C++/Onload SHARD backfill rejected (2026-09-10, 15:58 UTC): all 1,920,000 warmup and 1,920,000 measured requests complete, and all 32 sessions pass stability, but only shard 3 reports successful Logon. Shards 0/1/2 start on their requested cores without participation evidence. This is not a qualified four-active-listener measurement. The old auditor skipped C++/Onload admission and briefly imported an ok row; its numeric fields are now withdrawn, with the entire observation retained in the rejected backfill report. Both the parameter auditor and importer now require all C++/Onload owners to report Logon, just as for VMA. The cold check does not independently count each shard's application messages. The Onload listener-distribution cause remains under investigation; no transport setting or vendor library has been changed to hide this failure.

Onload shard launch correction (2026-09-10): the installed Onload build defaults EF_CLUSTER_SIZE to zero, disabling reuse-port clustering. The C++ FIX launcher did not override it. The launcher now derives the exact cluster size from --shards for a sharded server, keeps it zero for clients and non-sharded servers, and clears inherited cluster overrides. C++ and pure Rust share this bounded launch rule. The parameter auditor and importer require the matching server launch stamp as well as successful Logon on every listener; configuring a cluster alone cannot qualify the cell. No vendor library changes are involved.

Full-size C++/Onload SHARD validation completed (17:03 UTC): all four listeners now report successful Logon. All 32 sessions complete at the exact 50k/s aggregate target, with service p50/p99/p99.9/max 7.898 / 8.382 / 9.084 / 28.186 µs and response 8.639 / 9.543 / 10.027 / 29.486 µs. The matrix is populated. Service CV is 0.30% / 2.31% / 1.59%; startup affinity verifies server cores 10–13 and clients 15–18. The live watcher has 13 PASS, one INCONCLUSIVE, and no violations. Source and runtime remain frozen, and no FIX processes remain. This cold participation evidence is not a per-shard application-message counter. The parameter checker also notes that this C++ shard path lacks a bench_profile stamp, leaving that functional-profile readback unverified. The observation is therefore explicitly complete · audit warning, not a full parity pass. The stamp and the checker's all-profiles-absent bypass still need correction. The retained validation report preserves the original imported verdict and the reason for this qualification.

32-session Java MONO policy qualification (2026-09-10): the logged non-tiered / zero-method-trap-limit candidate now has a full kernel MONO control, without client phase tracing. All 1,920,000 measured requests complete at the unchanged 50k/s aggregate target (1,562.5/session); all 32 sessions pass the existing service stability thresholds, independently replayed from raw samples. This is not yet a qualified policy: the initial warmup sheds 8,685 requests, and one server application JIT event still occurs during measured run 1. Runs 2 and 3 have none. The original warmup evidence and all measured samples are retained. Startup placement and sampled live affinity pass: MONO server 10, clients 11/12/13/15, as configured (the 15–18 client set belongs to POOL). Source/runtime remain frozen and cleanup completes. See the kernel candidate review. The matching logged default-policy kernel control also passes all 32 sessions, with all 1,920,000 warmup and 1,920,000 measured requests delivered and no shedding. Its measured application JIT event counts are 12/0/0, all on the server. Default-policy service p50/p99/p99.9/max are 95.277/152.878/160.770/370.724 µs, versus the candidate's 100.037/162.563/173.951/389.466 µs. These cell summaries are maxima of the per-session three-run means, not pooled percentiles or absolute maxima. This pair does not demonstrate a kernel MONO improvement from the candidate, nor attribute the roughly 95–100 µs median to JIT activity.

The VMA candidate control completes all 1,920,000 measured requests, but still warns on two sessions (30 PASS / 2 WARN). It sheds 96,269 warmup requests; measured shedding, missing samples, stale replies, raw drops and clamps are zero. Its measured application JIT event counts are 1/0/0. Thus the passing SINGLE pair does not generalize to a fully stable MONO policy. Startup and sampled live affinity pass, with VMA maintenance on 14, and source/runtime remain frozen through cleanup. The matching logged default-policy VMA control also completes every measured request but warns on three sessions (29 PASS / 3 WARN), with 34,646 warmup requests shed. Its measured application JIT event counts are 46/0/0. Both policies retain median-variance warnings; the candidate does not cure them and has more warmup shedding in this pair. These are individual controls, not an estimate of warning probability. None of these logged diagnostic values replace canonical matrix rows or clear their partial status. The verification report retains the analysis scripts and checks raw evidence hashes, frozen source and runtime, diagnostic rejection and the generated comparison below.

What the original MONO partials mean: raw replay of the retained September 8 captures confirms complete measured delivery and the original warning classifications. Kernel has one warning session: its three service medians are 86.752/75.912/87.444 µs (sample CV 7.76%, above the 5% warning threshold), while its p99/p99.9 are stable. VMA has three median-variance warnings (CV 5.36–6.17%) and one p99.9 warning (31.541/20.949/20.747 µs, CV 25.29%, above 25%). These are different stability failures, not missing replies or evidence that every warning is a compiler pause. Those older runs have no JIT chronology and are not same-binary controls for the current policy experiment. Original samples, rows and warnings remain unchanged.

JNI optional-TLS build correction: the ordinary TCP proxy-aware dial helper was accidentally hidden inside an OpenSSL-only block. It is now available in both builds; the actual JNI acceptor/pool regression compiles and passes with TLS disabled and enabled. The normal benchmark build keeps TLS enabled; no benchmark transport or proxy policy was changed.

Java/VMA shard topology audit correction: the startup auditor now expects one server JVM for both Java providers, matching the harness's four-listener VMA topology. It also requires the explicit shard count, one acceptor thread per shard, exact worker-core list, multiplexing and reuse-port flags. Previously it rejected valid pure-Java launches and could accept JNI transcripts with no declared shard count. Actual worker and VMA-maintenance readbacks remain mandatory; this correction does not relax placement checks.

Kernel busy-poll isolation correction: mixed-transport harness runs now clear the kernel-only busy-poll flags before selecting each cell. Previously a kernel window could leak into subsequent native-backed VMA or Onload launches. The canonical OPEN sweep also clears ambient BUSY_POLL_USECS and LIBHFT_BUSY_POLL_USECS, retaining the committed machine-file default (including HP's single-session window). Direct bench.sh diagnostics may still set an explicit override. This was reproduced in inert routing/environment tests, not attributed as the cause of SCAN's earlier VMA stall, which had no window configured.

Earlier live-watcher qualification: sample 104 in open-v4-gcc14-20260908-9PgBhs/pinning-watch.log reports a violation for .NET threads pinned on housekeeping cores 0–9 during managed/kernel/single startup. The next sample accepts the same cores once each has two idle threads. Review found an inconsistent singleton/shared-thread placement check in the watcher. The retained log records the process name, not each thread's role, so those historical threads cannot be conclusively identified as GC workers. The corrected watcher reads each actual task name and allows only identified CLR server-GC workers on explicitly declared housekeeping cores; unknown off-isolated pins fail whether alone or shared. Positive CPU-delta collision checks remain in force. Its 19 inert regression tests pass, but the correction does not retroactively certify those historical thread identities. Startup measurement-core read-back passes remain separate evidence; this sitting does not claim an all-PASS live audit.

GCC 14 C++/kernel/mono window replay: all 1,920,000 measured exchanges reconcile across three 12.8-second intended phases. Full one-second windows contain 49,999–50,000 actual task starts and 49,997–50,001 completions. At the sampled window boundaries, generator backlog is zero and outstanding replies fluctuate between one and two; these boundary samples do not rule out shorter intra-window delays. Final drain takes 32.679–37.124 µs. The exact final scheduled slot lies at phase end and is included in the separate drain accounting. This supports sustained delivery of this 50,000/s workload, not a maximum-rate claim. The matrix row remains partial for run-to-run latency variance.

Pure Java/VMA fallback confirmed in the GCC 14 baseline: at 2026-09-08 13:58:07 UTC, the active mono cell's exclusive port 45758 had all 32 established connections in both namespaces' kernel tcp6 tables, using IPv4-mapped IPv6 addresses. The server was PID 3029780 and the first client PID 3031396. Java's default channel family is IPv6 on this host, while VMA 9.8.80's socket-offload path accepts only AF_INET. These are kernel-fallback measurements with VMA loaded, not accelerated VMA latency. The same launcher/channel configuration is used by the earlier pure-Java VMA single result; its historical socket table was not captured. JNI opens IPv4 sockets explicitly. The benchmark launcher now requests the IPv4 stack in each Java path, without changing production Java APIs. The correction is staged and local/installed launcher SHA-256 hashes match. The latest readiness-gated mono retry above confirms offload at all 64 endpoints and complete measured delivery; its timings remain partial for run-to-run instability. The original baseline was not changed mid-sweep. Direct access to root-owned FD links was unavailable; live attribution uses the exclusive cell port, namespace, endpoint addresses and exact launcher command lines. The first client's VMA statistics also showed no socket or ring blocks while the kernel flows were present. That one diagnostic reader ran on housekeeping cores during the cell; no latency effect is assumed absent. Retained observations and their limitations: java-pure-vma-family-20260908-JDgrEn under the external benchmark evidence root.

Two throughput benches

Target throughput: offer a fixed aggregate request rate, independently of replies, and report response latency, achieved rate, late sends and incomplete work. The standard target is 50 000 requests/s for the whole cell, not for every session.

Maximum throughput: sweep open-load rates to find the highest repeatable, sustained completed request/reply rate for the same topology and resources. Report the last passing / first failing rate bracket and response latency at the passing rate. The current closed-loop test is limited to one outstanding request per session; it does not establish this maximum.

The maximum-throughput view is not measured yet. Candidate rates can be run today, but an automated sustained-rate search, backlog validation and dedicated publication path remain to be implemented. Do not copy closed or fixed-target results into it.

taskset -c 0-9 python3 bench/load/run-open-matrix.py --host scan --schedule-version 4 --list

bench_target_root=$(mktemp -d /home/yann/libhft-bench-runs/target-throughput.XXXXXX)
taskset -c 0-9 python3 bench/load/run-open-matrix.py --host scan --schedule-version 4 \
  --out "$bench_target_root/sitting"

bench_max_root=$(mktemp -d /home/yann/libhft-bench-runs/maximum-throughput.XXXXXX)
env -i HOME="$HOME" USER="$USER" LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  BENCH_MACHINE=scan BENCH_STANDARDS="$PWD/bench/load/load-bench-standards.conf" \
  OPEN_SCHEDULE=cell-v4 LOOP_MODE=open WIRE=nos_er TPUT=100000 \
  WARMUP=20000 ITERS=3000000 RUNS=3 BENCH_RAW_SAMPLES=1 \
  taskset -c 0-9 bash bench/load/bench.sh --scenario single \
  --only '^cpp/kernel$' --out "$bench_max_root/rate-100000"