# hp C++ SocketXtreme pool at 100k: queue saturation and an intermittent lost flow Recorded 2026-09-20 on hp. This is an investigation record, not a published latency result and not a recommendation to deploy a tuner profile. The normal installed C++ SocketXtreme binary was restored to the profile-free build after the experiments. **Update, 2026-09-21.** The coordinator-to-worker completion hand-off has since been changed from a per-child mutex to a bounded SPSC queue while retaining single-caller VMA API exclusion. The fix removes the millisecond pool collapse; its ownership tests, artifact hash and two complete 50k/s controls are recorded in [`RECORD_2026-09-21_hp-socketxtreme-pool-spsc.md`](RECORD_2026-09-21_hp-socketxtreme-pool-spsc.md). The original intermittent one-flow failure remains historical evidence rather than being retrospectively attributed to that queue mutex. ## What future runs should know - A real 100,000-message/s C++/SocketXtreme/pool failure was captured, but it has not yet reproduced. One client flow stopped receiving replies; the open marker ring then correctly failed closed rather than overwriting an unanswered correlation slot. - No VMA poll error, socket error, NIC receive drop, or VMA orphan/refusal was logged in that failure. Do not relabel `live-marker-slot` as its root cause. - Independently, the current four-worker pool can fill every worker-owned completion queue to its 240-entry admission threshold (256 retained entries less one 16-event VMA poll batch). The coordinator must then stop draining the single shared VMA ring until workers consume enough packets. This produces multi-millisecond queueing even when every request completes at the requested rate. - The rejected `hp-cascadelake-gcc14-20260919` profile is unsafe to make the benchmark default. One actual tuned C++ binary control had a similar median but a 8.44 ms p99, against 118.5 us for the immediately preceding profile-free control. It was restored after the test. - Counter builds now print coordinator poll/routing totals and per-child queue high-water marks after a successful SocketXtreme pool run. They are diagnostic-only and must not provide publishable latency rows. ## Scope and exact recipes | Item | Value | | --- | --- | | Host | hp, Xeon Gold 6254; isolated application cores 8-17 | | Transport | VMA 9.8.80 SocketXtreme over mlxA/mlxB, `VMA_RING_ALLOCATION_LOGIC_RX=0` | | Pool placement | coordinator 8; workers 9-12; VMA internal threads 13; clients 14-17 | | Load | aggregate 100,000/s, 32 sessions, four client processes, cell-v4 open schedule | | Phase sizes | 15,625 warmup and 93,750 measured messages/session; 5 s + 30 s | | Original failing commit | `c233c7a0` | | Diagnostic source head | `a5eb8364` before the counter-output patch in this record | Every targeted control used: ```bash OPEN_SCHEDULE=cell-v4 TPUT=100000 RUNS=1 BENCH_RAW_SAMPLES=1 \ ONLY='^cpp/socketxtreme$' \ bash bench/load/bench.sh --scenario pool --out ``` `BENCH_RAW_SAMPLES=1` is the publication-shaped capture mode. The C++ client collects its measured vectors during the run in either mode; retained raw files are written after the phase, so raw-file I/O cannot explain a warmup failure. ## Captured failure The original retained output is `/home/yann/libhft-bench-runs/maximum-throughput-hp-20260920-tuned/pool`. Its recipe records raw capture, 100,000/s, 32 sessions, and the cell-v4 schedule. In client process 1, warmup run 1 failed after 1.564 s: ```text reason=live-marker-slot session=4 sent=38266 answered=37343 outstanding=923 session 2: sent=4271 answered=3759 inflight=512 shed=586 socket 2: borrows=3760 poll_failed=0 ``` The other seven sessions in that process received about 4,700-4,850 replies. There were no session errors, no send retries, and no poll failure. The special `0x100000000` event in the dump is `VMA_SOCKETXTREME_PACKET`, not an error. The capture establishes that a single flow stopped progressing; it does not establish whether the loss was in VMA, the queue hand-off, or another transient component. The later current-source controls completed all requested work, including one three-run, normal, profile-free run (9 million measured round trips). They do not disprove an intermittent fault: | Capture | Binary | Runs | p50 / p99 / p99.9 (us) | Delivery | | --- | --- | ---: | --- | --- | | `profilefree-normal-repeat1` | normal profile-free | 1 | 3,218 / 35,611 / 40,287 | complete | | `profilefree-normal-repeat2` | normal profile-free | 1 | 10,897 / 47,174 / 50,590 | complete | | `profilefree-normal-threepass` | normal profile-free | 3 | 33.5 / 104.0 / 156.3 | complete; stability PARTIAL | | `profilefree-normal-raw` | normal profile-free | 1 | 35.2 / 118.5 / 345.2 | complete | The installed profile-free normal binary was restored and verified as SHA-256 `086299a374f7ab48cda8aa3a25e48b84150fa5528890bf4cae90633bab018357`. ## Confirmed saturation mechanism The counter build intentionally adds timing/counter work, so its latency cannot be compared with normal rows. Its ownership evidence is nevertheless decisive: ```text listener: polls=20,375,490 polls_empty=17,629,650 routed=6,991,761 dropped=0 children: queue_peak=232..243, poll_failed=0 ``` Each child queue holds 256 completions. Before polling, `PollAndRoute()` requires room for an entire 16-completion batch in **every** child (`PumpHasRoom()`), so the effective shared-ring admission threshold is 240 unread entries. The observed 242/243 peaks show repeated global back-pressure. A retained entry is a VMA completion rather than necessarily one application request, so this counter must not be converted directly into an RTT; it does, however, explain the multi-millisecond queueing in the 36.4 ms median / 52.3 ms p99 counter run. This is a functional throughput/back-pressure design limit of the coordinated pool, not proof that it caused the separate one-flow loss. The queue is empty by normal shutdown, hence the need for a high-water mark rather than relying on the final queue length. An attempted coordinator `yield()` after routed bursts and during queue back-pressure was rejected: two raw-capture repetitions completed but had 29-31 ms medians and about 49 ms p99. Yielding the sole ring poller lets the VMA ring fall behind; do not restore that change without a redesigned pipeline. ## Tuned-profile control The old recipe names `hp-cascadelake-gcc14-20260919`, but its `tuned_kernels=` field probes managed shims only. It cannot attest the C++ benchmark executable, which has no equivalent runtime marker. A real tuned control was therefore rebuilt with: ```bash make -C engine/cpp \ LIBHFT_TUNED_DIR=tools/libhft-tune/profiles/hp-cascadelake-gcc14-20260919 \ LIBHFT_ALLOW_REJECTED_TUNED_PROFILE=1 vma_session ``` The temporary installed binary was SHA-256 `47dd39c4d0e7d64a791443d849f7b12ed602dc4fa424e400aa0af54b02599685`. It completed all 3 million measured replies but reported p50 35.57 us, p99 8,441.44 us, p99.9 13,246.57 us, and maximum 14,446.25 us. The immediately preceding profile-free raw control was 35.15 / 118.50 / 345.22 / 916.05 us. This is one paired sequential control, not a full tuner gate, but it agrees with the existing tuner verdict: that profile is byte-identical to the rejected candidate and must not become a default merely because it is machine-specific. It also means old records cannot be used to claim that C++ was tuned unless the actual artifact hash is retained. Two follow-up, one-run isolations show why it would be misleading to nominate one pick from this saturated transport as the cause: | Temporary tuned variant | p50 / p99 / p99.9 (us) | Interpretation | | --- | --- | --- | | Full profile | 35.6 / 8,441 / 13,247 | Full rejected prescription | | `FIX_MESSAGEIN_STRUCTURAL_PARSE=0` | 47.0 / 6,322 / 8,364 | Structural parser contributes but is not sufficient | | Full profile, `FX_FIND_BYTE_IMPL=FX_FIND_BYTE_SWAR` | 37,517 / 42,097 / 56,156 | Reverting the AVX2 scan is much worse at this offered load | All completed delivery. These are sequential saturation probes, not independent primitive benchmarks: a small service-time change moves the shared queue across its back-pressure cliff. They establish that the current full profile is not transferable, not a ranked list of deployable individual picks. The profile-free binary was restored after every temporary install. ## C++ compiled-profile provenance (2026-09-21) The provenance gap above is now closed for new C++ benchmark cells. Every `bench_roundtrip*` process writes one `bench_tuned_profile=` line after any VMA bootstrap re-exec. It contains the resolved compile-time selections for all profile-controlled primitives, including tag parsing, integer/decimal codecs, whole-field split, `FIX_MESSAGEIN_STRUCTURAL_PARSE`, and structural indexing. This is deliberately more than `tuned_build=0|1`: the rejected Cascade Lake candidate's decisive settings are now visible in the log as `structural_parse=1;struct_idx=FX_STRUCT_IDX_AVX2`. New recipes set `cpp_tuned_profile_log=1`. The runtime pinning/provenance audit requires exactly one complete stamp from the C++ server and every C++ client, requires them to agree, and rejects a tuned binary when the fingerprinted machine declaration remains profile-free (or the inverse). A tuned binary also carries the `cksum` fingerprint of the exact `libhft_tuned.h` used at compile time; the audit compares it with the recipe fingerprint. A machine declaration that names a profile marked `REJECTED` is refused before a cell starts. Older captures lack this schema marker and remain readable, but cannot retrospectively establish their C++ tuning selection. The profile-free SocketXtreme binary currently installed after this change is SHA-256 `5b9729967b428d22515e55d288e3a1238226298c8f298d2e7c601a8c9202297c`. Its emitted signature records `tuned_build=0`, loop tag parsing, SWAR byte search, JEAIII integer formatting, no structural parser, and sequential structural indexing. The corresponding installed Java and .NET Native shims emit the same full profile-free signature. A one-run C++/SocketXtreme/single cell was used only to verify the log/audit path; it is not a 100k pool result and is not publication evidence. ## Next work 1. Keep reproducing the exact raw-capture 100k control from a clean profile-free binary until the lost flow recurs; retain the first complete server/client logs, socket diagnostics, raw files and hardware counters. 2. Treat a low-latency 100k SocketXtreme pool as an architectural task. VMA's extra API has one ring owner; today workers also require that API lock to consume/free zero-copy packets and send replies. A safe improvement needs a measured ownership-preserving pipeline, not a lock/yield tweak. 3. Re-run autotuning only when its prescription or transfer gate changes, and retain its complete gate evidence. A new CPU generation alone is not proof that every per-primitive winner changes.