# SCAN — all-language bench run, 2026-08-11 C++, Java and .NET benches on SCAN, plus the compiler/runtime version sweeps. Includes the transport/Solarflare run for the same box and day (see the final section). (C++-only, and the tuner fix) and [`SCAN_BYPASS_RIG.md`](../SCAN_BYPASS_RIG.md) (Solarflare). Conditions: cores 18/19 isolated, C1E capped, microcode `0xb000040`, `irqbalance` off. Everything serialised — L3 is one shared 50 MiB instance across all 20 cores, so a concurrent build corrupts a measurement without interrupting it. Toolchains: g++ 11.5, clang 21.1.8, JDK 17.0.19 / 21.0.12 / 25.0.4, .NET SDK 8.0.129 and 10.0.302. ## Getting all three to build at all Three build blockers, all the same shape: a BEELINK2/WSL path hardcoded as an unconditional default. Each is fixed and committed. | Where | Symptom on SCAN | Fix | |---|---|---| | `bench/cpp/fixbench/Makefile` | `mkdir: cannot create '/mnt/c'` | probe for `/mnt/c`, else `./bin` | | `bench/dotnet/hft-bench/.../Libhft.Bench.csproj` | wall of `CS0246: CoreSession not found` | ProjectReference walked up 4 levels, needed 5 | | `bench/dotnet/fixbench/quickfix-n/...csproj` | wall of `CS0246: QuickFix not found` | probe known QuickFIX/n locations | ⚠️ The `.csproj` failures deserve note: **MSBuild demotes a missing `ProjectReference` to a warning** and keeps building, so the real error surfaces later as missing types and never names the path at fault. ## Java — JMH microbenches across JDK 17 / 21 / 25 Build and run interleaved per JDK: `engine/java` and `hft-bench` install into the single local Maven repo, so building for one JDK overwrites what another would resolve. Building all three upfront then running all three would measure the last JDK's bytecode three times. | benchmark | JDK 17 | JDK 21 | JDK 25 | 25 vs 17 | |---|---:|---:|---:|---:| | `scanSohFields` | 102.93 | 83.50 | 82.86 | **−19.5%** | | `findClOrdIdTag` | 52.49 | 45.01 | 44.77 | **−14.7%** | | `parseLongAscii` | 20.38 | 18.87 | 18.25 | −10.4% | | `formatChecksum3Ascii` | 10.12 | 9.60 | 9.59 | −5.3% | | `formatPositiveLongAscii` | 79.50 | 76.55 | 76.66 | −3.6% | | `parseDecimalScaled6` | 20.10 | 19.74 | 19.72 | −1.9% | | `checksumMessage` | 79.38 | 78.22 | 78.64 | −0.9% | ns/op, lower is better. **JDK 25 is faster than 17 on every row**, and the wins concentrate in the scan/search paths rather than the arithmetic ones — 21 gets most of it, 25 adds a little. ## .NET — QuickBench microbenches (net8.0) | benchmark | ns/op | alloc B/op | |---|---:|---:| | `checksum.avx2[64]` | 10.44 | 0 | | `checksum.avx2[256]` | 15.43 | 0 | | `checksum.avx2[1024]` | 42.24 | 0 | | `checksum.scalar[256]` | 158.59 | 0 | | `encode.nos` | 165.33 | 0 | | `encode.nos+sessionHeader` | 241.59 | 0 | | `frame.extract` | 229.15 | 0 | | `parse.fields` | 395.22 | 0 | | `roundtrip.encode+frame+parse` | 803.65 | 0 | **Zero allocations on every row.** AVX2 vs scalar checksum at 256 B is 10.3×. ## C++ — full 4-arm matrix: {g++ 11.5, clang 21.1.8} x {stock, tuned} All 28 binaries, one rep each, quiet box, core 19. | bench (ns) | g++ stock | g++ tuned | clang stock | clang tuned | |---|---:|---:|---:|---:| | `ParseBuffer` | 355.26 | 316.99 | 506.09 | 470.41 | | `ParseExtract` | 358.46 | 323.20 | 507.25 | 487.65 | | `GenLogon` | 99.32 | 99.10 | **43.11** | 43.11 | | `GenNewOrderSingle` | 147.22 | 144.57 | **48.61** | 48.64 | | `GenExecutionReport` | 140.42 | 143.94 | **53.54** | 53.54 | ### The compilers split the workload, and it is real **clang is ~3x faster on message GENERATION and ~1.4x slower on PARSING.** That is large enough to suspect a benchmark artefact, so it was checked against instructions retired, which a timer cannot fake: | `GenNewOrderSingle` | ns/op | ins/op | branches | IPC | |---|---:|---:|---:|---:| | g++ stock | 147.22 | 732 | 146 | 2.26 | | clang stock | 48.61 | **279** | **48** | 2.60 | | `ParseBuffer` | ns/op | ins/op | IPC | |---|---:|---:|---:| | g++ stock | 355.26 | 2,076.8 | 2.65 | | clang stock | 506.09 | **2,763.6** | 2.48 | Instruction counts move **with** time in both directions — fewer on generate, more on parse — and IPC stays healthy in all arms, so nothing was optimised away into a fake win. Dead-code elimination would not make clang do *more* work on the parse path. ⚠️ One question stays open: the generation benches encode the same message every iteration, so part of clang's advantage could be hoisting loop-invariant work a real workload could not. That does not apply to parsing (data-dependent), and it does not explain the parse regression, but it should be settled before acting on the generation figure. **This matters because the tuner selects one compiler for the whole build** (`LIBHFT_TUNED_CXX := g++`). A FIX engine both parses and generates, so a single choice is leaving one side on the table either way. Per-translation-unit compiler selection is worth investigating — or, better, working out why g++ needs 732 instructions where clang needs 279 for the same source. ## C++ — tuned profile vs stock (g++ 11.5) Median of 3 runs per arm, core 19, full rebuild between arms. | bench | stock | tuned | Δ | |---|---:|---:|---:| | `ParseBuffer (5-msg rotate)` | 355.41 | 319.51 | **−10.1%** | | `ParseExtract (5-msg rotate)` | 356.06 | 325.81 | **−8.5%** | | `GenLogon` | 99.65 | 98.76 | −0.9% | | `GenNewOrderSingle (D)` | 147.22 | 146.73 | −0.3% | | `GenExecutionReport (8)` | 141.12 | 144.99 | +2.7% | ns, lower is better. The tuned profile is a real win on the parse paths and neutral on generation — consistent with the tuner's own picks, which changed `find_byte`, `int_parse` and `parse_mode` but kept the generation kernels. ## Cross-language FIX round-trip ⚠️ **Not yet a valid comparison.** Recorded here so the trap is not re-walked. **Now valid**, after the core-allocation fix described below. | engine | p50 | vs C++ | note | |---|---:|---:|---| | C++ `bench_roundtrip` | 16.4 µs | 1.00x | | | Java **JNI** provider | **22.2 µs** | 1.36x | `scheduler_delay` p50 0.22 µs | | Java pure-managed | 33.0 µs | 2.01x | `scheduler_delay` p50 0.29 µs | | .NET Pure | 42.2 µs | 2.57x | | ### ⚠️ CORRECTION: the .NET Pure figure of 42.2 µs was MISMEASURED The 42.2 µs .NET round trip that framed this whole exercise was taken with `taskset -c 18` for the server and `-c 19` for the client — **one core per process**. That is the same core-starvation mistake diagnosed (and written up at length) for the Java lane on this very page. Re-measured with four cores per process, no code change: | .NET Pure | p50 | p99 | p99.9 | worst | stability | |---|---:|---:|---:|---:|---| | 1 core/process (original) | 42.17 | 46.17 | — | 8,771 | FAIL | | **4 cores/process** | **19.92** | 31.44 | 2,836 | 7,954 | FAIL | **2.1× faster from core allocation alone.** So .NET Pure was never 1.9× behind Java/JNI — it is level with it (22.2 µs) and well ahead of Java pure-managed (33.0 µs). ### .NET Pure: GC and JIT tuning transforms the tail ⚠️ **Corrected**: an earlier revision credited the GC knobs. Isolating them shows the GC settings do **nothing** here and the entire fix is `TieredCompilation=false`. | isolated knob | p50 | p99.9 | worst | stability | gen0 | |---|---:|---:|---:|---|---:| | `TieredCompilation=0` only | 22.08 | **27.11** | **46.51** | **PASS** | 0 | | GC settings only | 22.50 | 1,811.69 | 6,339.97 | FAIL | 0 | **`gen0 = gen1 = gen2 = 0` in every run** — no collection ever happens, and the whole process allocates ~1.1 MB. The tail was never GC; it is **JIT re-tiering**, tier-0 code being replaced mid-run. The tail *looks* like GC, which is why the wrong lever was credited first. Now set in `LibhftNet.Bench.csproj` rather than requiring env vars, verified with no environment at all: **p50 21.08, p99 22.60, p99.9 25.57, worst 46.69, stability PASS**. Original measurement, same binary, same cores: | .NET Pure | p50 | p99 | p99.9 | worst | stability | |---|---:|---:|---:|---:|---| | default runtime | 19.92 | 31.44 | 2,836 | 7,954 | **FAIL** | | **tuned runtime** | 21.50 | **22.96** | **26.36** | **46.92** | **PASS** | - **p99.9: 2,836 → 26.4 µs — 108× better** - **worst: 7,954 → 46.9 µs — 170× better** - run-to-run stability goes FAIL → **PASS** ### .NET Pure: removing a clock read from the measured wait `Peer.ProcessUntilCount` called `Stopwatch.GetTimestamp()` on **every spin iteration** — a vDSO call inside the measured wait for the ExecutionReport, purely to service a timeout that essentially never fires. Now checked every 256 turns; the timeout can overshoot by at most 256 `ProcessOne()` calls, which is microseconds against a millisecond timeout. | tuned runtime | p50 | p99 | p99.9 | worst | |---|---:|---:|---:|---:| | before | 21.50 | 22.96 | 26.36 | 46.92 | | **after** | **20.85** | **22.29** | **25.35** | **45.97** | ~3% off p50 and every percentile improves, reproducible across two reps. ### Where .NET Pure actually stands | config | p50 | p99.9 | worst | stability | |---|---:|---:|---:|---| | original (1 core/process) | 42.17 | — | 8,771 | FAIL | | 4 cores, default runtime | 19.92 | 2,836 | 7,954 | **FAIL** | | 4 cores, default + clock fix | 21.49 | 2,037 | 8,094 | **FAIL** | | **4 cores, tuned + clock fix** | **20.85** | **25.35** | **45.97** | **PASS** | ⚠️ The 19.92 µs figure is **not a reproducible floor** — it came from a `stability=FAIL` run, and a repeat of the same configuration produced 21.49 µs. Only the tuned configuration yields `PASS` and repeatable numbers, so 20.85 µs is the defensible result. **Net: 42.17 → 20.85 µs p50 (2.02×), worst 8,771 → 45.97 µs (191×), FAIL → PASS.** At 20.85 µs .NET Pure is now *faster* than Java/JNI (22.2 µs) and 1.58× faster than Java pure-managed (33.0 µs). The GC trade for a trading engine: the default configuration's tail was multi-millisecond, and a 2.8 ms p99.9 is a dropped order, not a slow one. The gen0 budget is the likely lever — QuickBench reports zero allocations per op on the codec path, so the pauses come from the harness/session layer rather than the hot codec. ### .NET Native provider: parity target MET | engine | round-trip p50 | vs Java/JNI | |---|---:|---:| | **.NET Native (hftnet)** | **16.4 µs** | **0.74x — faster** | | Java/JNI | 22.2 µs | 1.00x | | Java pure-managed | 33.0 µs | 1.49x | | .NET Pure | 42.2 µs | 1.90x | Three reps: 17.10 / 16.40 / 16.21 µs, spread 5.5%, `lost=0` and `sendFail=0` in all three. The native provider closes the 1.9x gap and goes past it. ✅ **The echo caveat is resolved.** The responder now encodes a fresh ExecutionReport (35=8 with OrderID/ExecID/ExecType/OrdStatus/Symbol/Side/ LeavesQty/CumQty/AvgPx/TransactTime) instead of echoing the inbound frame, so the round trip carries the same work as Java's `nos_er`: parse a NOS, then build and send a distinct ER. Three reps with the real responder, n=5000 each, zero loss: | rep | p50 | p99 | |---|---:|---:| | 1 | 16.25 | 17.34 | | 2 | 19.48 | 20.69 | | 3 | 16.28 | 17.23 | Median **16.28 µs**, versus Java/JNI's 22.2 µs. Encoding a real ER did not move the number materially (16.71 → 16.28), which says the responder side was never where the time went. ⚠️ Remaining differences, smaller but real: n=5000 per rep against 15,000 for the Java runs, and this is a purpose-built harness rather than the shared hft-bench machinery the Java figures come from. The responder is also a load responder, not a conformant acceptor — ClOrdID is not echoed back, since a tag lookup on the hot path would change what is being measured. ⚠️ Also unresolved: runs of ~5,500 iterations lose replies while 550 do not, so something degrades with session age. 500-sample runs are clean; longer ones are not yet trustworthy. **Original gap:** .NET was **1.9x behind Java/JNI** — the gap the native provider has to close. Managed-to-managed, .NET is 1.28x behind pure Java, so there is headroom even before native code. ⚠️ All three Java/.NET rows report `stability=FAIL` on run-to-run variance (cv_p50 0.11 for JNI, 0.26 for pure). The magnitudes are credible and the providers now separate cleanly; the tails are not yet trustworthy. The Java provider harness run reported `end_to_end` p50 = 26,001 µs with `scheduler_delay` = 25,903 µs. Those being near-identical means the load generator was saturated and the number is **backlog, not latency** — classic coordinated omission. The harness itself flagged `INSUFFICIENT` (one run). A re-run with `-Dhftbench.runs=3 -Dhftbench.iterations=20000` exceeded 500 s. ### The Java round-trip run was MISCONFIGURED BY ME — not a broken lane ⚠️ **Correction.** An earlier revision of this doc concluded the lane "reports harness pacing, not engine latency" and told readers not to use it. That was wrong, and the reasoning behind it was wrong twice over. Both errors are kept here because each is an easy one to repeat. **What I observed.** Both providers returned byte-identical figures: | provider | `end_to_end` p50 | `scheduler_delay` p50 | |---|---:|---:| | `pure-java` | 26001.407 | 25903.103 | | `jni-native` | 26001.407 | 25903.103 | **Wrong explanation #1 — "coordinated omission back-fill".** I proposed that `recordValueWithExpectedInterval` was synthesising samples and swamping the engine signal. The harness prints `Correcting for coordinated omission: false` and reports `corrected_count = 0`. No correction was applied. Retracted. **Wrong explanation #2 — "identical medians prove the number is not a measurement".** Also wrong. The 26 ms is a perfectly real elapsed time. **The actual cause.** The harness said so plainly, and I did not read its own warning: ``` Status: complete, end_to_end 10000/10000, deficit 0, achieved 38.256/s, WARN achieved throughput 38.256/s below 95% of target 10000/s ``` 38.256/s is 26.14 ms per iteration — exactly the reported p50. **I pinned the run to two cores (`taskset -c 18,19`). The reference configuration in `LIBHFT_PURE_JAVA_FULL_PLAN.md` is `taskset -c 2-7` — six.** This bench runs a busy-spin initiator worker *and* a busy-spin acceptor worker, plus the measurement thread and the JVM's own threads. `initiatorWorkerCore` and `acceptorWorkerCore` both default to `-1` (unpinned), so on a two-core mask the spinners starve each other and every round trip waits for a scheduler timeslice. That is also why the two providers agreed to three decimals: both were measuring my scheduling stall, which dwarfs any engine difference — and at ~26 ms with `significantValueDigits = 3` the histogram bucket is ~26 µs wide, so a provider-level difference could not have shown up even if present. **The lane is fine. The operator was not.** Correct invocation needs at least six cores and explicitly pinned worker cores: ```bash taskset -c 10-17 java -cp "$CP" \ -Dlh.provider.tcp.provider=pure-java \ -Dlh.provider.tcp.initiatorWorkerCore=18 \ -Dlh.provider.tcp.acceptorWorkerCore=19 \ -Dhftbench.throughput= \ org.latency.libhftjava.LHProviderTcpHftBench ``` Until that is re-run, there is still no valid Java-vs-.NET round-trip ratio — but for a mundane reason, not because the tooling is untrustworthy. **Lesson: when a harness prints a WARN about its own achieved throughput, that is the finding.** I built an elaborate theory on top of a number the tool had already flagged as invalid. The .NET number carries its own caveat: `stability=FAIL` with an 8.8 ms worst case against a 42 µs p50 means the tail is not yet trustworthy either. ## Not run / blocked - **Six-arm wire matrix** (kernel/Onload/TCPDirect) — needs one `sudo install` into the root-owned bench dir. - **Full C++ 4-arm matrix** ({g++, clang} × {stock, tuned}, 7 binaries) — started and deliberately stopped after 5 binaries so the box could be freed for parity work. Needs a clean re-run on a quiet box. - **`benchutils` exits rc=195 — diagnosed, not a failure.** Its `main` ends with ```c return (int)(g_sink32 ^ (g_sink64&0xffffffffULL)); ``` It returns the anti-optimisation **sink** as the process exit status, which the OS truncates to 8 bits. rc=195 is leftover XOR data, not an error. The benchmark itself is fine. It still needs fixing: any CI gate or script that checks exit codes reads it as a failure, and the value moves with the workload, so it is an intermittent false alarm rather than a stable one. `bench_getymd.cpp` has the same pattern (`return (int)g_sink;`). The fix is to keep the sink observable — a `volatile` read the compiler cannot elide — while returning 0. - **QuickFIX/n comparison arm** — builds now, not yet run. --- ## ⚠️ The six-arm run below does NOT meet the benchmark contract See [`CONTRACT_OPERATIONAL.md`](CONTRACT_OPERATIONAL.md). The arms differed in offered load (C++ unthrottled at 43,751/s vs Java paced at 5,000/s — 8.75×), sample count (2,000 to 60,000 — 30×) and topology (one-process vs two-process). **The claim ".NET Native is level with C++" does not survive that** and is retracted. What survives: QuickFIX/n is 50–70× slower than everything else (no confound of that size is plausible), and Java Pure vs Java/JNI is a clean comparison — identical harness, load, topology, JVM and pinning, differing only in the engine. Corrected worst-case column, which an earlier revision showed as `—` because the output lines were truncated at 125 characters rather than because the data was missing: | Engine | p50 µs | p99 µs | p99.9 µs | worst µs | |---|---:|---:|---:|---:| | C++ | 16.52 | 17.61 | 19.71 | **31.59** | | .NET Native | 16.64 | 19.14 | 37.97 | **96.34** | | .NET Pure | 21.47 | 22.98 | 28.20 | **92.90** | | Java/JNI | 21.58 | 26.10 | 40.77 | **87.30** | | Java Pure | 36.06 | 64.95 | 128.81 | **1,420.01** | | QuickFIX/n | 1,140.01 | 1,237.88 | 1,812.75 | **10,505.08** | Java Pure's worst is **15× .NET Pure's** — material, and hidden by the truncation. ## Full engine bench — all six arms, one driver, 2026-08-11 Every engine on SCAN, same box, same day, same scenario (`nos_er`), serialised through one script so conditions cannot drift between arms. Cores 10–19 isolated; **≥4 cores per process** throughout — one core per process is the starvation mistake that cost a 2.1× error on .NET and 788× on Java earlier in this document. | Engine | p50 µs | p99 µs | p99.9 µs | worst µs | stability | |---|---:|---:|---:|---:|---| | **.NET Native (hftnet)** | **16.20** | 17.42 | — | — | n=5000, 0 lost | | **C++ `bench_roundtrip`** | **16.28** | 17.46 | 22.23 | — | 20k samples, 44.1k msg/s | | **.NET Pure** | 21.47 | 22.98 | 28.20 | 92.90 | **PASS** | | **Java/JNI** | 21.63 | 26.47 | 38.72 | — | WARN | | **Java Pure** | 34.78 | 65.44 | 116.44 | — | **FAIL** | | **QuickFIX/n** | 1,140.01 | 1,237.88 | 1,812.75 | — | pass | ### What this run establishes 1. **.NET Native is level with the C++ reference** — 16.20 vs 16.28 µs. The C ABI costs nothing measurable at session granularity, which is the whole premise of the coarse-operation design (per-primitive P/Invoke measured a net *loss*: 27.6 ns vs .NET's own managed AVX2 at 15.4 ns). 2. **.NET Pure matches Java/JNI** — 21.47 vs 21.63 µs. A pure managed engine is level with Java crossing JNI into C++. 3. **.NET Pure beats Java Pure by 1.6×** — the managed-vs-managed comparison. 4. **QuickFIX/n is 53× slower** than .NET Pure, reproducing the earlier 1,142.80 µs within 0.2%. ### Reproducibility Every arm reproduced its earlier independent measurement closely: .NET Native 16.28 → 16.20, .NET Pure 21.08 → 21.47, Java/JNI 22.20 → 21.63, Java Pure 33.00 → 34.78, QuickFIX/n 1,142.80 → 1,140.01. Nothing here rests on a single sample. ### Caveats that still stand - **Java Pure reports `FAIL`** and Java/JNI `WARN` on run-to-run variance; .NET managed is the only multi-run arm at `PASS`. - **Topology is not uniform.** C++ and QuickFIX/n run both roles in one process; .NET Pure runs two processes on separate core sets; .NET Native runs acceptor and initiator in one process. Same box and scenario, different harness shapes — which is why the 16.20 vs 16.28 tie should be read as "no measurable ABI cost", not as a ranking. - The .NET Native arm has **no p99.9/worst** because its harness reports p50/p99 only. ## Java tail: not GC, and not JIT either Java's run-to-run variance (pure `FAIL`, JNI `WARN`) had never been explained. Two hypotheses tested on SCAN, both refuted. **GC — measured, and there is none.** | Provider | achieved | p50 µs | p99 µs | stability | GC events | |---|---:|---:|---:|---|---:| | Java Pure | 4,994/s | 36.06 | 64.95 | FAIL (cv 0.239) | **0** | | Java/JNI | 4,992/s | 21.58 | 26.10 | WARN (cv 0.061) | **0** | `-Xlog:gc` confirms the logging engaged (`Using G1` at 0.003s) and the JVM collected **nothing** across 3 runs × 5,000 iterations plus warmup at ~5,000 msg/s. Both engines are genuinely allocation-free on the hot path. **JIT re-tiering — tested because it was the answer for .NET. It is not the answer here.** | Provider | cv_p50 baseline | cv_p50 with `-XX:TieredStopAtLevel=4` | |---|---:|---:| | Java Pure | 0.239 FAIL | 0.255 FAIL | | Java/JNI | 0.061 WARN | 0.057 WARN | Unchanged. p50 shifted slightly (pure 36.06 → 33.40) but that is *inside* the variance being measured — a cv of 0.25 means p50 itself swings ±25% run to run. ⚠️ **The two managed runtimes do not share a cause.** .NET's multi-millisecond tail WAS JIT re-tiering (`TieredCompilation=false` took p99.9 from 2,836 µs to 27 µs). Assuming Java would behave the same was reasonable and wrong. **Best remaining discriminator:** Java/JNI (cv 0.061) and Java Pure (cv 0.239) share the same transport, harness, JVM and pinning, and differ only in the engine. A 4× variance gap across that boundary points at pure-Java's own code rather than at the socket, the scheduler or the harness. --- ## Contract-compliant scenario matrix — SCAN, 5,000/s First matrix run under [`CONTRACT_OPERATIONAL.md`](CONTRACT_OPERATIONAL.md): every arm paced to 5,000 msg/s, isolated cores, ≥4 cores per role, serialised. p50 µs. | scenario | C++ | .NET Native | .NET Pure | Java/JNI | Java Pure | QuickFIX/n | |---|---:|---:|---:|---:|---:|---:| | `nos_er` | 17.05 | **16.55** | 21.21 | 21.40 | 33.95 | 1,141.28 | | `md` | 18.96 | *no support* | 27.61 | 22.27 | 30.51 | 1,156.55 | | `md_nos_er` | 35.80 | *no support* | 48.98 | 43.30 | 61.80 | 2,212.97 | | `tick_to_trade` | 16.55 | *no support* | — | 0.000 ⚠️ | 0.000 ⚠️ | 1,159.73 | C++ detail (5,000 samples each, all achieving 5,000 msg/s): | scenario | p50 | p99 | p99.9 | |---|---:|---:|---:| | `nos_er` | 17.05 | 19.66 | 26.70 | | `md` | 18.96 | 21.75 | 31.70 | | `md_nos_er` | 35.80 | 38.48 | 52.69 | | `tick_to_trade` | 16.55 | 26.24 | 32.14 | `md_nos_er` is ≈2× `nos_er` for **every** engine — the expected shape for combined market-data + order flow, and a useful sanity signal that the scenarios do what they claim. ### ⚠️ Not results - **Java `tick_to_trade` = 0.000 with `PASS`.** A zero median that passes stability is not a measurement. The C++ arm reports `tick_to_trade: no samples` on the other three scenarios, so **the probe wiring is suspect across engines**, not just in Java. - **`order_round_trip` and `tick_to_fill` report `no samples` on every C++ scenario.** Two probes are wired and never produce data. Nobody had noticed because the previous extraction discarded those lines. - **.NET Pure `tick_to_trade`** produced no p50. - **.NET Native covers 1 of 4** — its harness has no `--scenario`. ### The extraction bugs, recorded because they repeated Four separate reporting defects, each sitting on top of a run that had *succeeded*: 1. `cut -c1-125` truncated the `worst` column → reported as `—`, hiding that Java Pure's worst is 1,420 µs (15× .NET Pure's). 2. `substr($0, RSTART+8, ...)` was off by one → every p50 lost its leading digit (`21.208` → `1.208`, `1141.278` → `141.278`). 3. `grep ... | tail -1` kept the trailing `tick_to_trade: no samples` line and **discarded the aligned-RTT result** → the C++ column was empty for three scenarios, and the data was never written to the log at all. 4. A single-manifest grep concluded "no internal-harness dependency" while `java_fixbench_report.py` resolved `a third-party benchmark library` from a runtime-generated POM. **The measurements were fine every time; the extraction destroyed them.** Verify the reporting path against one known-good raw line before trusting any table built from it. --- ## Full matrix, second contract-compliant run Reproduces the first run closely (`nos_er`: .NET Pure 21.17 vs 21.21, QuickFIX/n 1144.73 vs 1141.28, Java/JNI 21.86 vs 21.40), so these are stable figures, not single-sample artefacts. p50 µs at 5,000/s. | scenario | C++ | .NET Pure | QuickFIX/n | Java/JNI | Java Pure | |---|---:|---:|---:|---:|---:| | `nos_er` | 17.05 | 21.17 | 1,144.73 | 21.86 (WARN) | 34.84 (FAIL) | | `md` | 18.96 | 27.85 | 1,152.02 | 22.16 (WARN) | 33.47 (FAIL) | | `md_nos_er` | 35.80 | 50.27 | 2,214.75 | 44.37 (WARN) | 59.68 (FAIL) | | `tick_to_trade` | 15.84 | — | 1,158.29 | 0.000 ⚠️ | 0.000 ⚠️ | ### Rows that are NOT results - **`.NET Native` is absent.** Porting it to the shared harness exposed a p99 of 4.3 ms and a worst of 9 ms behind a steady 16.8 µs median (`cv_p50 = 0.005`, `stability = FAIL`). Its earlier ~16 µs rows came from a single-run hand-rolled loop with no coordinated-omission alignment and understated the tail. **Excluded until the tail is found.** - **Java `tick_to_trade` = 0.000 with `PASS`** — a zero median that passes stability is not a measurement. - **.NET Pure `tick_to_trade`** produced nothing. ### The C++ column was discarded a second time The driver still contained `grep -E 'aligned RTT|tick_to_trade' | tail -1`, which keeps the trailing `tick_to_trade` line and throws away the RTT result. I diagnosed this defect, fixed it in a **separate** `cpp_all.sh`, and never patched the driver — so it reproduced exactly. The C++ figures above come from that dedicated run; the driver is now fixed to `grep -E '\[client\].*p50='`. **Lesson: fixing a bug in a copy is not fixing the bug.** This is the fifth extraction defect in this session and the only one to recur, because it was the only one patched somewhere other than where it lived. ## .NET Native's 9 ms tail is a BETWEEN-RUN artefact, not a steady-state defect | runs | n | p50 | p99 | p99.9 | worst | |---:|---:|---:|---:|---:|---:| | **1** | 5,000 | 16.32 | **18.13** | **21.99** | **41.15** | | 3 | 15,000 | 16.82 | 4,343.17 | 8,672.93 | 9,006.28 | A single run is clean. The milliseconds appear only at `runs=3`, so the engine's steady state is sound and something in the **run transition** costs ~9 ms, with those samples landing in the aggregate. This narrows the hunt considerably and exonerates the engine: - **Not** the acceptor spin loop or the 500,000-iteration bounded wait — both are per-iteration and would show inside a single run. - **Likely** the task's `BeforeRun`/`RunComplete` path, or session state carried across runs (the initiator and acceptor are created once and reused, so run 2 starts against a session that has already been driven 5,000 times). ⚠️ It remains a real defect for reporting: a 3-run aggregate is what the contract requires, so the arm's published tail *is* the 9 ms one until this is fixed. But "steady-state p99 is 18 µs and run transitions cost 9 ms" is a materially different statement from "this engine has a 9 ms tail", and only the first is supported. --- ## tick_to_trade is a SERVER-side probe — I was reading the wrong role `.NET Pure / tick_to_trade` was recorded as producing nothing, and Java's as `0.000 with PASS`, and both were called dead probes. Neither is dead. ``` server: scenario=tick_to_trade samples=15000 runs=3 p50=18.000us p99=20.381 p999=24.718 worst=39.805 stability=PASS client: samples=0 ... stability=INSUFFICIENT ``` `tick_to_trade` measures the **server's** market-data-to-order reaction span, as `CONTRACT_SCENARIOS.md` states under *Server Tick-To-Trade Scenarios*. The client legitimately has zero samples: it does not measure that span. I read the client aggregate for every arm and concluded the probe was broken. **.NET Pure `tick_to_trade` = 18.00 µs p50, PASS** — the cell is filled. This very likely applies to the Java `0.000 (PASS)` cells too: a zero median that passes stability is exactly what a client-side aggregate of a server-side probe looks like. ⚠️ **Checked: NOT recoverable from disk for Java.** The existing `tick_to_trade` sections contain only `end_to_end` probe rows — the Java arm never emitted a `tick_to_trade` probe row in that configuration. Its `0.000` has the same root cause in a different form: the extractor read `end_to_end`, which correctly has zero samples in a tick-to-trade run. The cells need a re-run with the right probe captured, not a re-read. So the fix applied to `.NET Pure` (read the server aggregate) does **not** transfer to Java (whose harness is single-process and emitted no such row). One cell recovered, not five. It probably also explains the C++ arm printing `tick_to_trade: no samples` on the other three scenarios: in `nos_er`/`md`/`md_nos_er` there is no tick to react to, so a server-side reaction probe having no samples is correct behaviour, not a defect. Same for `order_round_trip` and `tick_to_fill` outside their own scenarios. **Lesson: a probe reporting no samples may be the wrong ROLE, not a broken probe.** Check which side of the pair owns the measurement before calling it dead — the scenario contract says so per scenario. ## The ~25 ms fixed cost is SHARED, not engine-specific Java `tick_to_trade` reports a huge `scheduler_delay`. Testing whether it is load-driven settles it — it is not: | offered rate | `scheduler_delay` p50 | |---|---:| | 5,000/s | 25,804 µs | | 1,000/s | **25,002 µs** | **A 5× rate reduction moved it 3%.** That is a fixed cost, not starvation. ⚠️ **It is the same magnitude as Philadelphia's 26.8 ms fixed cost per iteration** — a different engine, a different language, the same ~25–27 ms. Two independent "engine defects" showing an identical fixed cost points at something shared: the harness's tick/feed generator, or a fixed wait in the scenario driver, rather than either engine. **Investigate the shared tick source before either engine.** Chasing them separately — as both were queued to be — would mean debugging the same cause twice, in two languages, and probably concluding twice that the engine is fine. ### RESOLVED: the shared cause is that thread pinning never happened at all The "shared cause" call above was correct, and the shared thing is not a tick source. It is **`libhftbench.so`, which no build target ever produced.** Every hft-bench role affinity — `-Dclient.core`, `-Dserver.core`, and the harness's own worker roles — goes through `NativeThreadAffinity`, whose static initialiser calls `System.loadLibrary("hftbench")`. With no such library, that threw, `AVAILABLE` latched to `false`, `bindCurrentThread` returned `false` forever, and `AffinityBinding` **discarded the request and continued**, printing one line to *stdout*: ``` hft-bench philadelphia-client-io affinity unavailable; requested=18 ``` which sat directly beneath four JVM `WARNING:` lines about restricted methods and was read as more of the same. Nothing pinned anything. Every busy-spin thread floated across the cpuset, contending with the measurement thread, and what got measured was the scheduler. **Measured, same command, sole variable being whether the `.so` exists:** | arm | | achieved | `end_to_end` p50 | `scheduler_delay` p50 | |---|---|---:|---:|---:| | Philadelphia (fast) | broken | 37.3/s | 36,044 µs | 26,804 µs | | Philadelphia (fast) | **fixed** | **4,927/s** | **23.5 µs** | **0.169 µs** | | Java `tick_to_trade` jni-native | broken | — | 25,804 µs | 25,903 µs | | Java `tick_to_trade` jni-native | **fixed** | — | **10.0 µs** | **0.199 µs** | | Java `tick_to_trade` pure-java | **fixed** | — | **14.5 µs** | **0.255 µs** | **1,472× on Philadelphia's latency and 101× on its throughput.** Philadelphia is not a slow engine; it was never given a core. **Why this was hard to see, and what actually found it.** The failure had no error, no exception, and no missing output — every run completed with `Status: complete`, `deficit 0`, full sample counts and a `PASS` stability verdict. `cv_p50` was **0.000000**, because an unpinned run is *reproducibly* unpinned; the run-to-run stability gate certified the wrong number as trustworthy. What exposed it was reading the log's non-result lines rather than grepping straight to the metrics, which is where the `affinity unavailable` line had been sitting all along. Contributing errors of mine, both now in the contract's repeated-mistakes table: 1. **I declared the pinning hypothesis refuted on a run that could not have tested it.** Setting `-Dclient.core=18` under `taskset -c 10-17` was itself invalid — core 18 is outside the process mask — and the properties were being dropped regardless. "Pinning made no difference" was a measurement of nothing. 2. **I assumed a property that is accepted is a property that is applied.** The knob parsed, logged its request, and did nothing. **Fixes.** `bench/java/hft-bench/build_native.sh` builds the library (it needs `bench/cpp/hft-bench/include` for `HftBenchPlatform.h`). `AffinityBinding` now **aborts** when a named core cannot be bound, with `-Dhftbench.affinity.allowUnavailable=true` as a loud opt-out. `HftBenchRuntime` installs a fatal uncaught-exception handler so a dead bench thread halts the JVM (exit 70) instead of letting the survivors report a number. **Scope of invalidation.** Every pure-Java arm — Philadelphia above all — and every scenario relying on harness-level pinning. libhft's own Java providers pin their worker threads *inside the C++ engine* via JNI, so `nos_er` provider figures were never affected by this. C++, .NET Pure (`taskset`) and .NET native (`--acceptor-core`, a direct `sched_setaffinity`) were never affected. All affected arms are being re-measured as matrix v4. ### The achieved-rate gate does not apply to server-side scenarios The same run reports `achieved 0.000/s`, because achieved rate is computed from `end_to_end` samples and a `tick_to_trade` run has none by design. So the contract's "verify achieved rate before reading any latency" check is **inapplicable** to server-side scenarios and must not be read as a failure there. `tick_to_trade` needs its own liveness gate — sample count against expected, not achieved rate. ## tick_to_trade cells filled — and an accounting inconsistency corrected I was counting Java `nos_er` = 34.84 µs (FAIL) as a filled cell while treating `tick_to_trade` FAIL rows as unfilled. Both come from the same harness with 15,000 samples over 3 runs and a reported status. That was inconsistent, and it understated the matrix. Server-side reaction span, 5,000/s, 15,000 samples, 3 runs: | engine | p50 µs | p99 | p99.9 | stability | |---|---:|---:|---:|---| | **Java/JNI** | **10.15** | 13.16 | 21.90 | FAIL | | **Java Pure** | **11.16** | 26.87 | — | FAIL | | .NET Pure | 18.00 | 20.38 | 24.72 | **PASS** | | C++ | 15.84 | 26.24 | 32.14 | single-run | | QuickFIX/n | 1,159.73 | 1,570.31 | 5,153.32 | pass | Both Java rows carry `FAIL` on run-to-run variance and are reported with that status, exactly as the `nos_er` rows are. The `scheduler_delay` fixed cost documented above is a **client-side** pacing artefact; `tick_to_trade` measures the **server's** reaction span and is not derived from it. Ordering worth noting: on this scenario Java/JNI (10.15) and Java Pure (11.16) are *faster* than .NET Pure (18.00) and C++ (15.84) — the reverse of every round-trip scenario. That is a real difference in what the probe measures (reaction span, not round trip) and should not be read as a contradiction of the round-trip ordering. **Matrix: 16 of 32.** ## QuickFIX/J: the NPE was a working-directory problem `QFHftBench` died with `NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null`, which I recorded as a missing-session-config defect. The real cause is upstream and simpler — two `ConfigError` lines precede it: ``` quickfix.ConfigError: qf_acceptor.cfg (No such file or directory) quickfix.ConfigError: qf_initiator.cfg (No such file or directory) ``` QuickFIX/J resolves those **relative to the working directory**, and they live in `bench/java/fixbench/config/`. Run from there and it works. The NPE is QuickFIX/J's own downstream failure after the config load returns nothing — the exception that surfaced was three steps past the actual error, and I read the last line instead of the first. **Run it from `bench/java/fixbench/config/`.** ### Filled cell, at a sustainable rate ``` achieved 786.651/s against an 800/s target (within 2%) nos_er: n=9000 runs=3 p50=245.759us p99=49,921.578 p99.9=71,068.330 FAIL ``` ⚠️ **Not comparable to the 5,000/s matrix rows.** QuickFIX/J cannot sustain 5,000/s — its ceiling is ~1,200/s — so it was run at 800/s. **This exposes a flaw in the matrix design: the rate must be set by the slowest arm.** A matrix at 5,000/s silently excludes every engine that cannot reach it, which is exactly the engine a comparison most wants to include. Either the whole matrix runs at a rate all arms sustain, or each row states its own rate and the table stops pretending to be a single comparison. **Matrix: 17 of 32** — counting this row as filled-but-rate-qualified. ### QuickFIX/J: the rate ceiling is SCENARIO-dependent Running `md`, `md_nos_er` and `tick_to_trade` at the 800/s that worked for `nos_er` — all three fail the rate gate: | scenario | target | achieved | valid? | |---|---:|---:|---| | `nos_er` | 800/s | 786.7/s | ✅ within 2% | | `md` | 800/s | 457–667/s | ❌ WARN, below 95% | | `md_nos_er` | 800/s | 361–620/s | ❌ WARN, below 95% | | `tick_to_trade` | 800/s | no probe row emitted | ❌ | The reported p50s (251.9 µs for `md`, 542.7 µs for `md_nos_er`) are **queueing, not latency**, and are not counted. Same rule that invalidated the earlier Java and Philadelphia runs. So a single per-engine rate is not sufficient either — **the ceiling varies by scenario within one engine**, roughly 790/s for `nos_er`, 450–670/s for `md`, 360–620/s for `md_nos_er`. A matrix wanting valid QuickFIX/J rows everywhere needs ~300/s, at which point every other arm is running two orders of magnitude below its own capability. **That is the real shape of the problem: there is no single rate at which all eight arms produce a valid, meaningful row.** The matrix should either publish per-cell rates explicitly, or accept that a competitive table spanning a 50× capability range cannot be one table. **Matrix: 17 of 32** — unchanged; these three attempts produced no valid cell. ### QuickFIX/J `md`: four attempts, and where I stopped | target | achieved | % of target | gate | |---:|---:|---:|---| | 800/s | 457–667/s | 57–83% | ❌ | | 300/s | 279/s | **93%** | ❌ (needs 95%) | Converging, so ~200/s would likely pass and fill the cell. **I stopped there deliberately.** At 200/s the row would sit in a table beside C++ at 5,000/s, and its p99 is already 43 ms. A cell filled at 1/25th the rate of its neighbours is not a comparison; it is a number that looks like one. Filling it would satisfy a cell count while making the table less honest — which is the failure the measurement policy exists to prevent. **What should be recorded instead is the capability**: QuickFIX/J sustains ~790/s on `nos_er` and cannot hold even 300/s on `md`. That is a more useful statement about the engine than any p50 obtained by lowering the rate until the gate stops complaining. **Matrix: 17 of 32**, and the remaining QuickFIX/J cells are marked *rate-infeasible* rather than pending — they are not blocked on effort. ## Path to the remaining 15 cells Concrete next action per blocker, in cost order. Each is stated so it can be started without re-deriving anything. ### A. Philadelphia + Java `tick_to_trade` — 5 cells, ONE shared cause Both show a fixed ~25–27 ms independent of offered rate (Java: 25,804 µs at 5,000/s vs 25,002 µs at 1,000/s — a 5× rate change moved it 3%). Two engines, two languages, one constant ⇒ the harness tick/feed generator, not either engine. **Action:** find the tick source in `bench/java/hft-bench` and the scenario driver. Look for a fixed sleep/park or a feed interval near 25 ms. Confirm by changing it and observing both numbers move together. **Searched and eliminated (2026-08-12)** — the constant is *not* a simple literal: - `parkNanos` sites in `LHProviderTcpHftBench`: all 100 µs, and all in logon/bind wait loops, not the measured path - `sendIntervalNanos`: derived from `expectedIntervalNanos()`/target throughput, so it tracks the configured rate — it cannot produce a rate-independent constant - `defaultTickToTradeSampleTimeoutNanos`: `max(100ms, min(1s, interval×1000))` = 200 ms at 5,000/s — a timeout, and the wrong magnitude - grep for `25_000_000` / `25000000` / `TICK_INTERVAL` / `tickIntervalNanos`: no hits So it is **not a hardcoded 25 ms anywhere obvious**. Next candidates, untested: an emergent period (a wait-strategy backoff reaching a plateau), a socket-level timer such as delayed ACK interacting with the feed, or something in `BenchRunner`'s inter-run pause (`pauseAfterWarmupMillis`) leaking into the measured phase. **Measure before assuming**: log the delta between consecutive tick timestamps and see whether 25 ms is one wait or an accumulation of smaller ones. **Why first:** one fix, five cells, and it retires two separately-queued hunts. ### B. .NET Native run-transition tail — 4 cells Steady-state p99 is 18 µs; ~9 ms appears at run boundaries with `cv_p50 = 0.005` (periodic, not load). Ruled out already: GC (0 collections), JIT tiering, load, pinning. **Action:** instrument `BeforeRun`/`RunComplete` and session reuse across runs — the cost is at the transition, so the suspect is per-run setup/teardown, not the hot path. Then add `--scenario` (typed sends already exist in ABI 7+). ### C. QuickFIX C++ two-process deadlock — 4 cells Hangs with a peer present; exits cleanly alone with `client logon timeout`. No core, so not memory safety. **Action:** `gdb -p` on **both** hung processes simultaneously. A handshake deadlock shows one side blocked on a read the other never sends. Re-run with `stdbuf -oL` first — the empty logs may be lost buffering, and the last line before the hang localises it immediately. ### D. QuickFIX/J `md` / `md_nos_er` / `tick_to_trade` — rate-infeasible, ~2–6 cells Not blocked on effort. Either publish with an explicit per-cell rate and the capability note, or leave empty and cite the ceiling. **Do not lower the rate until the gate passes** — see the section above. ### Design decision this needs There is no single rate at which all eight arms produce a valid row (~50× capability spread). Before more cells are chased, decide: **one table with per-cell rates stated**, or **separate tables per capability class**. Filling cells without that decision produces a table that reads as one comparison and is not. ### Blocker A narrowed: `scheduler_delay` is a fixed SCHEDULE OFFSET | run | interval | `scheduler_delay` p50 | |---|---:|---:| | 5,000/s, default wait | 200 µs | **25,804.799** | | 5,000/s, **busyspin** | 200 µs | **25,804.799** | | 1,000/s | 1,000 µs | 25,001.983 | **Wait strategy eliminated** — busyspin reproduces the value byte-for-byte. Two structural facts: 1. **Byte-identical to three decimals across independent runs.** That is a constant, not a measurement — the same tell that exposed the Java provider issue earlier in this document. 2. The two values differ by **802.8 µs** against an **800 µs** interval difference, giving ``` scheduler_delay ≈ 26,005 µs − interval ``` So this is a **fixed ~26 ms offset between the harness's schedule origin and the first actual start**, not a per-iteration cost. The loop begins ~26 ms behind `next` and never recovers — which is exactly why no rate change and no wait strategy moves it. ⚠️ `BenchRunner`'s measured loop contains `if (taskStart > next) next = taskStart`, a catch-up that should erase precisely this offset. **Either that line is not reached on the `tick_to_trade` path, or `next` is re-seeded from a stale origin between runs.** That is the next thing to read — a much smaller target than "the harness". This also means `scheduler_delay` on this scenario is **not evidence of core starvation**, and the earlier reading of it as such was wrong. The engine is not starved; the schedule origin is stale. ### Blocker A: candidate LOCATED in `BenchRunner` ```java long next = System.nanoTime(); // seeded HERE OsJitterMonitor.start(config, context); // <-- setup runs between RuntimeTelemetry.capture(...); // the seed and the loop long measuredStart = System.nanoTime(); context.startMeasuredPhase(measuredStart); final BenchContext.Probe schedulerDelayProbe = ...; for (long i = 0; i < config.measuredIterations(); i++) { // loop starts HERE next += interval; ``` `next` is seeded **before** `OsJitterMonitor.start()` and `RuntimeTelemetry.capture()`. Whatever those cost, `next` is already that stale when the loop begins — a **fixed, rate-independent offset**, which is precisely the shape measured (`scheduler_delay ≈ 26,005 µs − interval`). The warmup loop (line ~119) has the same seeding pattern and **no catch-up line at all**, unlike the measured loop. **Fix to test:** re-seed `next = System.nanoTime()` immediately before the `for`, after all setup. One line. ⚠️ **Not yet proven.** The measured loop does contain `if (taskStart > next) next = taskStart`, which should absorb the offset after the *first* iteration and leave p50 ≈ 0 — yet p50 is 25.8 ms, meaning most samples carry it. So either that catch-up does not execute on this path, or `next` is being re-staled each iteration. **Verify by logging `next` and `taskStart` for the first ten iterations before changing anything** — if only iteration 1 shows the offset, this candidate is wrong and the cause is elsewhere. ### Blocker A RESOLVED (diagnosis): `runIteration` takes ~26 ms The stale-seed candidate above is **wrong**. The catch-up line *is* on the measured path (`BenchRunner.java:160-161`, immediately after the sample), so a one-off stale seed would be absorbed after iteration 1 and leave p50 ≈ 0. The arithmetic identifies it instead. With the catch-up working, `next = taskStart_prev + interval`, so if the task itself takes `D`: ``` scheduler_delay = D − interval ``` | rate | interval | predicted (D = 26,005 µs) | measured | |---|---:|---:|---:| | 5,000/s | 200 µs | 25,805 | **25,804.799** | | 1,000/s | 1,000 µs | 25,005 | **25,001.983** | Both fit within 3 µs. **`runIteration` for `tick_to_trade` takes ~26 ms.** And `tick_to_trade` itself measures only ~11–15 µs — so **99.9% of that 26 ms is spent outside the measured span**, in the tick-generation / reaction-matching around it. **`scheduler_delay` was reporting correctly all along.** It said the loop cannot keep up, which is true: a 26 ms task cannot be paced at 200 µs. Every reading of it as core starvation — including several in this document — was wrong. The engine is fine; the scenario task is slow. **Next:** time the phases inside `tick_to_trade`'s `runIteration`. The measured span is 15 µs of 26 ms; find what the other 25.985 ms is waiting for. Philadelphia's identical fixed cost is very likely the same thing, since it runs the same harness. ## Matrix state — recounted, 21 of 32 I had been reporting 17, carried forward from before the `tick_to_trade` row was completed and QuickFIX/J's `nos_er` landed. Actual state, p50 µs: | scenario | C++ | .NET Pure | Java Pure | Java/JNI | QuickFIX/n | QuickFIX/J | filled | |---|---:|---:|---:|---:|---:|---:|---:| | `nos_er` | 17.05 | 21.17 | 34.84 | 21.86 | 1,144.73 | 245.76¹ | **6** | | `md` | 18.96 | 27.85 | 33.47 | 22.16 | 1,152.02 | — | **5** | | `md_nos_er` | 35.80 | 50.27 | 59.68 | 44.37 | 2,214.75 | — | **5** | | `tick_to_trade` | 15.84 | 18.00 | 11.16 | 10.15 | 1,159.73 | — | **5** | ¹ at 800/s; every other cell at 5,000/s. Rate stated per the design decision above. **21 of 32 (66%).** The 11 empty cells: | Arm | Cells | Status | |---|---:|---| | .NET Native | 4 | run-transition tail; also needs `--scenario` | | QuickFIX C++ | 4 | two-process deadlock, unconfirmed cause | | QuickFIX/J `md`/`md_nos_er`/`tick_to_trade` | 3 | **rate-infeasible** — not effort-blocked | So of 11 empty cells, **3 cannot be meaningfully filled at all** and 8 are blocked on two investigations. `tick_to_trade` is complete for every arm that can run it. ### Blocker C narrowed: the hang is in INITIALISATION, before any output Re-ran the pair with `stdbuf -oL -eL` to rule out lost buffering. **Both logs are still completely empty.** The client run *alone* prints `client logon timeout` normally, so output works — these processes genuinely produce nothing. That places the deadlock **before either process's first log line**, i.e. in session/transport setup, not in the measured loop. It also explains the absent core: nothing crashes, both sides simply block. Ruled out so far: memory safety (no core, clean solo exit), lost buffering (`stdbuf` unchanged), and the measured loop (never reached). **Config resolution — TESTED AND REFUTED.** Re-ran from the repo root: the harness writes both `quickfix_cpp_acceptor_stage5.cfg` and `quickfix_cpp_initiator_stage5.cfg` successfully, and the pair *still* hangs with empty logs. So this is not the QuickFIX/J failure mode repeating. Cumulative eliminations for this arm: memory safety (no core, clean solo exit), lost buffering (`stdbuf` unchanged), the measured loop (never reached), and now config resolution (files written, still hangs). **What remains:** the block is between config write and first log line — socket bind/connect or QuickFIX session start. The only instrument that will localise it is `gdb -p` attached to **both** hung processes simultaneously, reading each stack. Everything cheaper has now been tried and eliminated. ### Blocker C DIAGNOSED: the harness's own `wait_until`, not QuickFIX `gdb -p` on both hung processes. Identical shape on each: ``` Thread 2 poll() <- FIX::SocketMonitor::block <- FIX::SocketServer::block <- FIX::SocketAcceptor::onStart (server) poll() <- FIX::SocketMonitor::block <- FIX::SocketConnector::block <- FIX::SocketInitiator::onStart (client) Thread 1 sched_yield() <- (anonymous namespace)::wait_until( std::function const&, int) ``` **QuickFIX itself is healthy on both sides** — its I/O thread is in the normal `poll()` loop, acceptor accepting and initiator connecting. Nothing is deadlocked in the library. **The stage5 harness's main thread is spinning in its own `wait_until` on a predicate that never becomes true.** Both sides sit there, so neither ever observes itself logged on. That is why: no core (nothing crashes), no output (the harness logs *after* the wait), and a clean solo exit (alone, the wait times out and reports `client logon timeout`). **Fix direction:** find `wait_until` in the stage5 harness and the predicate it is given for the logon gate. The likely causes are a logon-callback flag never set — the QuickFIX `onLogon` callback not wired to the flag the predicate reads — or a session config where neither side initiates. QuickFIX is connecting, so the transport is fine; the signal from library to harness is what is missing. **All four cheaper hypotheses were eliminated first** (memory safety, buffering, measured loop, config resolution), so this stack was the necessary instrument rather than the first guess. ### Blocker C, final narrowing: past logon, stuck awaiting reports Source read of `quickfix_cpp_stage5_xmachine.cpp`: - `onLogon` **is** wired (line 247) and sets `logged_on_` with release ordering - `wait_until` (line 307) has a real deadline — `Clock::now() + timeout_ms` — so it cannot spin forever - the logon gate is lines 531/663: `wait_until([&]{ return app.logged_on(); }, args.timeout_ms)` - the **other** `wait_until` sites (493/498/504) await `reports_received()` / `snapshots_received()` The solo client exits cleanly with `client logon timeout`, proving the deadline works. So with a peer present the pair is **past logon** and blocked on one of the message waits — the client waiting for ExecutionReports that never arrive, and the server waiting on its own gate. **Next action, precise:** check the default of `args.timeout_ms` (a long default would explain an 80 s+ hang that never printed), then add a log line before each `wait_until` so the failing gate identifies itself instead of blocking silently. The predicate to instrument first is `reports_received() >= target_reports` at line 493. **Why the messages do not flow is the remaining question** — not why the process hangs. That is now understood. ### Blocker C ROOT CAUSE: logon never completes, both sides Running with `--timeout-ms 500` instead of the 20,000 ms default makes it print: ``` client: client logon timeout server: server logon timeout ``` **Both sides time out at the logon gate.** My previous narrowing ("past logon, blocked on a message wait") was wrong — they never get past logon at all. The 20 s default is what disguised this as a deadlock: with 110 iterations each waiting 20 s, nothing printable happens inside any reasonable observation window, so it looked like a hang. It was a fast, clean failure hidden behind a long timeout. **The defect:** TCP connects — gdb showed the acceptor accepting and the initiator connecting on healthy `poll()` loops — but the FIX logon handshake never completes. That is a session-config mismatch between the two generated `quickfix_cpp_*_stage5.cfg` files: CompIDs, `BeginString`, or the `--data-dictionary` they reference. **Config verified CORRECT — hypothesis refuted.** Diffed both generated files: - `SenderCompID`/`TargetCompID` properly mirrored (`CLIENT↔SERVER`) - `ConnectionType` initiator/acceptor, same port - `BeginString=FIX.4.4` on both - `UseDataDictionary=Y`, `DataDictionary=/home/yann/code/quickfix/spec/FIX44.xml` — **file exists at exactly that path** (315 KB, verified) - `StartTime==EndTime==00:00:00` — QuickFIX's 24-hour-session convention, not a zero-length window Nothing in the session config is wrong, so the cause is **not** configuration. **Cumulative eliminations for this arm — all tested, none assumed:** memory safety, lost buffering, the measured loop, config-file resolution, the logon predicate wiring (`onLogon` is wired), `wait_until` having no deadline (it has one), and now session-config correctness. **What remains:** both sides reach the logon gate and neither sees `onLogon` fire, with TCP demonstrably connected. Candidates: the harness never calls `initiator.start()`/`acceptor.start()` on this path, or starts them after the wait begins; or the two processes bind sessions that do not match on a field not covered above. **Instrument `onCreate`/`toAdmin`/`fromAdmin` — if the Logon message is never sent, the session was never started; if it is sent and refused, the reject text names the field.** **Debuggability fix worth making regardless:** the 20 s default timeout turns every failure on this arm into an apparent hang. Lower it, or log each `wait_until` entry, so failures announce themselves. ### Blocker B: two explanations eliminated by reading + arithmetic **Stale pacing seed — eliminated.** `.NET BenchRunner.DriveIterations` seeds `long next = start` at entry, freshly per run, so the offset bug hypothesised (wrongly) for the Java side does not exist here. **"First iteration of each run" — eliminated by arithmetic.** 3 runs means 3 first-iterations out of 15,000 samples = the 99.98th percentile. The observed degradation is at **p99 = 4.3 ms**, which needs ~150 affected samples, i.e. ~50 per run. So it is not a single post-transition sample; roughly the first 1% of each run is slow. That shape — a slow *prefix* of each run rather than a single spike — points at warm-up state being lost between runs and re-acquired over ~50 iterations. Candidates: TCP congestion window collapsing during the inter-run gap (`StopAccepting` → `ProbeStats` → `RunResult` → `RunComplete` is real wall time with no traffic), a cold branch predictor / instruction cache, or an engine-side buffer that refills lazily. **Next action, precise:** dump per-iteration latency with its index and check *where in the run* the outliers fall. If they cluster in the first ~50 iterations of each run, it is warm-up loss and the fix is either a per-run warmup the harness discards, or keeping traffic flowing during the inter-run gap. If they are scattered, it is something periodic and the index distribution will show the period. This is one measurement, and it discriminates every remaining candidate. --- # Full matrix v2 — .NET Native included, 24 of 32 p50 µs, 5,000/s paced, 3 runs × 5,000 iterations, isolated cores, ≥4 per role, serialised. Supersedes the earlier tables. | scenario | C++ | .NET Native | .NET Pure | Java/JNI | Java Pure | QuickFIX/n | |---|---:|---:|---:|---:|---:|---:| | `nos_er` | 17.06 | **16.68** ✅ | 21.78 | 21.78 ⚠️ | 46.30 ❌ | 1,143.55 | | `md` | 19.00 | 22.86 ✅ | 27.14 | 22.44 ❌ | 31.60 ❌ | 1,153.45 | | `md_nos_er` | 35.46 | 41.60 ✅ | 50.51 | 43.05 ⚠️ | 84.29 ❌ | 2,213.69 | | `tick_to_trade` | 16.44 | 15.41 ✅ | 18.10 | **10.02** ❌ | 13.53 ❌ | 1,158.17 | ✅ PASS ⚠️ WARN ❌ FAIL on run-to-run variance. ### .NET Native: blocker B is closed All four scenarios, `n=15000`, **`stability=PASS` on every one**, `cv_p50 ≤ 0.002`, zero lost, and **no 9 ms tail anywhere** — worst case 42.6–78.5 µs. The run-transition defect that had this arm excluded is gone, resolved by the incoming harness port onto `BenchRunner` plus the extraction-sink gate (`1359ee4`, `f832453`, `d1e8e36`). It was fixed while I was characterising it — worth recording that the fix was the *port*, which is what the earlier analysis concluded was the right move for reasons unrelated to this tail. **.NET Native is the only arm passing stability in all four scenarios.** Java pure fails all four; Java/JNI is mixed. ⚠️ **Do not quote the C++ comparison yet.** .NET Native beats C++ on `nos_er` (16.68 vs 17.06) and `tick_to_trade` (15.41 vs 16.44) but trails on `md` (22.86 vs 19.00) and `md_nos_er` (41.60 vs 35.46). A wrapper beating what it wraps on two of four scenarios is the same signal that turned out to be a workload asymmetry last time. Check the per-scenario field sets before publishing. ### QuickFIX/J — one cell measured, at its own rate `nos_er` = **245.76 µs** at 800/s (achieved 786.7/s, within 2%), n=9000, 3 runs, FAIL on variance. Omitted from the v2 run because it cannot hold 5,000/s, but the cell is measured and stands with its rate stated. ### Matrix total: 25 of 32 | Arm | Empty | Blocker | |---|---:|---| | QuickFIX C++ | 4 | **ROOT CAUSE FOUND — see below.** Earlier note kept for the record: **Root of the diagnostic difficulty was** the harness installs `NullLogFactory` and `NullStoreFactory` (`quickfix_cpp_stage5_xmachine.cpp:121,523`), so QuickFIX logs *nothing* — the library is silent by construction, which is why seven hypotheses had to be tested indirectly. **One-line diagnostic: swap `NullLogFactory` for `FIX::ScreenLogFactory` and rebuild.** The Logon exchange then prints, and either no Logon is sent (session never started) or a Reject prints and names the offending field. | | QuickFIX/J `md`/`md_nos_er`/`tick_to_trade` | 3 | **Rate-infeasible** — 360–670/s achievable against any useful target. Not effort-blocked; no rate makes them comparable. | **Realistic ceiling is 29 of 32.** The three QuickFIX/J cells are a property of the engine, not a gap to close — filling them would mean running at ~1/25th the rate of their neighbours, which produces a number that looks like a comparison and is not. ## QuickFIX C++ ROOT CAUSE: server receives the order, never replies Swapping `NullLogFactory` for `FIX::ScreenLogFactory` and raising the timeout to 3,000 ms makes the wire visible: ``` client → 8=FIX.4.4|35=A|49=CLIENT|56=SERVER|141=Y Logon server → 8=FIX.4.4|35=A|49=SERVER|56=CLIENT|141=Y Logon ← handshake COMPLETES (Logon contains ResetSeqNumFlag=Y, resetting sequence numbers to 1) client → 8=FIX.4.4|35=D|...|11=QFCPP-XM-NOS-1|38=1000000|44=1.2345|55=EUR/… ← server RECEIVES the NewOrderSingle ``` **Logon works. Orders flow. The server never sends `35=8` back.** So the previous diagnosis ("logon never completes") was **wrong** — logon simply takes longer than the 500 ms I had used to force a fast failure. That probe was too aggressive and produced a misleading `logon timeout` on both sides. **The defect is server-side application logic**: the harness's `fromApp` / ExecutionReport path does not respond to `35=D`. Not a session, transport, config or QuickFIX problem — all four are now demonstrably working. **Next:** read the server's `fromApp` in `quickfix_cpp_stage5_xmachine.cpp` and find why a NewOrderSingle produces no ExecutionReport. The client is waiting on `reports_received() >= target_reports` (line 493), which is exactly consistent. **Diagnostic changes to revert:** `NullLogFactory` → `FIX::ScreenLogFactory` at two sites (backup at `/tmp/qf_stage5_backup.cpp`). Better: keep them behind a `--fix-log` flag, since their absence cost eight rounds of indirect diagnosis. ### QuickFIX C++: narrowed to two candidates in `fromApp` `fromApp` (line 254) is correct on inspection, and the role flag is right: ```cpp WorkflowApplication app(true, args.scenario); // line 522, server WorkflowApplication app(false, args.scenario); // line 649, client if (server_role_ && value == FIX::MsgType_NewOrderSingle) { ... if (is_tick_to_trade_scenario(scenario_)) return; // <-- candidate 1 FIX::Message er = make_execution_report(seq); send_to_target(er, session_id); // <-- candidate 2 return; } ``` The server *does* receive `35=D` (confirmed on the wire) and `server_role_` is `true`, so control reaches this block. Only two things can then swallow the reply: 1. **`is_tick_to_trade_scenario(scenario_)` returns true for `nos_er`** — an early `return` before the ExecutionReport is built. Would happen if scenario parsing failed and `scenario_` defaulted to a tick-to-trade value. 2. **`send_to_target` fails silently** — no throw, no log, message never queued. **Next: log `scenario_` and the `send_to_target` return at that call site.** One run distinguishes them, and both are one-line fixes. Note the existing comment at line 321 already documents a related failure — a grouped `35=W`/`35=V` being session-rejected *before* `fromApp` when the data dictionary is unset, which "hangs md/md_nos_er". That is the same class of silent-swallow, and suggests this harness has a pattern of failures that produce no output rather than an error. ### QuickFIX C++: the silent-swallow mechanism identified **Candidate 1 eliminated by reading.** `is_tick_to_trade_scenario` is a strict `scenario == "tick_to_trade"` compare, and `require_canonical_scenario` throws on any unknown value — so `scenario_` cannot silently default and the early return cannot fire for `nos_er`. **Candidate 2 confirmed as the mechanism.** `send_to_target` returns `bool`: ```cpp bool send_to_target(FIX::Message& msg, const FIX::SessionID& session_id) { try { return FIX::Session::sendToTarget(msg, session_id); } catch (const std::exception& ex) { std::cerr << "sendToTarget exception: " ... ; return false; } } ``` but the call site ignores it: ```cpp FIX::Message er = make_execution_report(seq); send_to_target(er, session_id); // return value discarded return; ``` An *exception* prints to `cerr`. A plain `false` return prints **nothing**. That matches the observed behaviour exactly: the server receives `35=D`, builds the ExecutionReport, fails to send, and reports nothing. `FIX::Session::sendToTarget` returns `false` when it cannot route the message — typically the session is not found for that `SessionID`, or the message header carries CompIDs that conflict with the session being routed to. **Two-line fix, and it is worth making regardless of whether it resolves the benchmark:** ```cpp if (!send_to_target(er, session_id)) std::cerr << "send_to_target FAILED for ExecutionReport seq=" << seq << " session=" << session_id.toString() << "\n"; ``` Then the next run either reports the failure and names the session, or sends successfully and the arm works. **A benchmark harness that discards the return of its send path cannot report why it produced no data** — which is the whole of this arm's eight-round diagnostic history in one line of code. ### QuickFIX C++: send SUCCEEDS — the fault is on the client receive path Applied the return check and ran it. **Zero send failures logged.** ``` server: send_to_target FAILED ... → 0 occurrences ``` So `FIX::Session::sendToTarget` returns true: the ExecutionReport **is** sent. Candidate 2 is eliminated along with candidate 1. **Established, end to end:** 1. Logon completes both ways (wire-confirmed) 2. Client sends `35=D`; server receives it (wire-confirmed) 3. Server builds the ExecutionReport and **sends it successfully** (return checked) 4. Client's `reports_received()` never reaches its target → the wait times out **The fault is therefore in the client's inbound path**, between the socket and `reports_received()`. The client's handler is `if (!server_role_ && value == FIX::MsgType_ExecutionReport)` at line 279. **Prime suspect, and the harness documents this exact failure mode at line 321:** a message rejected by the data dictionary *before* `fromApp` is ever called. That comment describes grouped `35=W`/`35=V` being session-rejected on a duplicate tag, which "hangs md/md_nos_er". If `make_execution_report` emits a field the FIX44 dictionary rejects, the client's `fromApp` never runs and the counter never moves — matching every observation. **Next: log `fromAdmin` and `onMessage`/session-reject on the CLIENT.** If a `35=3` Reject appears, its `58=` text names the offending field in the ExecutionReport. The return check is **kept** — a benchmark that discards its send result cannot explain its own missing data, and it costs nothing when the send succeeds. ## QuickFIX C++ FIXED — `ExecType_FILL` is invalid in FIX 4.4 The client's Reject named the field exactly: ``` 35=3 58=Value is incorrect (out of range) for this tag 371=150 (tag ExecType) 372=8 (in ExecutionReport) 373=5 (value incorrect for this tag) ``` `make_execution_report` set `FIX::ExecType_FILL` — `'2'`, which was **removed in FIX 4.4**. FIX44 represents a fill as `ExecType_TRADE` (`'F'`). The dictionary rejected every ExecutionReport *before* `fromApp`, so `reports_received()` never moved and the client timed out. **Fix — one enum value:** ```cpp msg.setField(FIX::ExecType(FIX::ExecType_TRADE)); // was ExecType_FILL ``` **Result — the arm works:** ``` engine=quickfix_cpp scenario=nos_er status=pass samples=100 runs=1 p50_us=92.176 p99_us=154.667 p999_us=156.777 deficit=0 rejects=0 ``` ### What this arm cost, and why Eight successive characterisations, seven wrong: *segfaults → crash unconfirmed → hangs → hangs in initialisation → harness `wait_until` → past logon, awaiting reports → logon never completes → send fails*. Every one was corrected by a **better instrument**, never by better reasoning. The single reason it took eight rounds: **`NullLogFactory`**. The harness silenced QuickFIX entirely, so a protocol-level rejection — which FIX reports precisely, with the offending tag number in the message — was invisible. One run with `ScreenLogFactory` produced the answer immediately. **Two changes worth keeping permanently:** 1. A `--fix-log` flag enabling `ScreenLogFactory` on demand. 2. The `send_to_target` return check (already kept) — a benchmark that discards its send result cannot explain its own missing data. **Matrix: 26 of 32.** `nos_er` complete for every engine. The remaining QuickFIX C++ scenarios should now run; the 3 QuickFIX/J cells stay rate-infeasible. ## ⚠️ `ExecType='2'` is emitted by libhft's own responders in ALL THREE runtimes The QuickFIX C++ fix generalises, and this is the more important finding. FIX 4.4 **removed** `ExecType` values `'1'` (Partial fill) and `'2'` (Fill); a fill is `'F'` (Trade). We have direct proof — a real FIX44 dictionary rejected 14 consecutive ExecutionReports with `371=150 373=5`. `ExecType='2'` is emitted in: | Location | What | |---|---| | `bench/cpp/fixbench/src/libhft/bench_roundtrip.cpp:960,985` | **C++ bench responder** | | `engine/dotnet/src/native/hftnet_session.cpp:1029` | **.NET Native responder** | | `engine/java/src/main/native/hftjava_jni.cpp:3013,4033` | **Java/JNI** | | `…/quickfix_cpp_stage4_workflow.cpp:128` (×2) | QuickFIX harnesses | **These pass our benchmarks only because dictionary validation is off.** Against a FIX 4.4 counterparty that validates — precisely the QuickFIX client that rejected 14 in a row — every one of these ExecutionReports would be rejected. ⚠️ **Not fixed here, deliberately.** Changing `'2'` → `'F'` across three engines is a wire-behaviour change that alters bench payloads (message length changes, so checksum and possibly latency shift). It needs its own verified change, not a tail-end edit. The one instance already fixed is the QuickFIX stage5 harness, where it was blocking the arm entirely. **Why our benches never caught it:** they run responder and initiator from the same codebase with validation disabled, so an invalid value round-trips happily. The defect only surfaces against a *third-party* validating counterparty — which is the strongest argument yet for keeping QuickFIX in the matrix as a conformance oracle rather than only as a performance foil. **Action:** audit all `FIX_TAG_ExecType` writes against the FIX 4.4 dictionary, change `'2'` → `'F'` (and `'1'` → `'F'` with `LeavesQty>0` if partial fills are emitted anywhere), then re-run the matrix — payload length changes, so the numbers must be re-measured, not carried over. ## ⚠️ Validation asymmetry: QuickFIX validates, libhft does not The QuickFIX arms run with `--data-dictionary` (`UseDataDictionary=Y`); the libhft arms run without validation. That is **more application work inside the measured span** for QuickFIX — the conformity axis in §8 — and the matrix has been charging it for work its competitors do not do. Measured on the same binary, `nos_er`, 800/s, 100 iterations: | configuration | p50 µs | p99 µs | |---|---:|---:| | with FIX44 dictionary | 92.18 | 154.67 | | **without dictionary** | **68.22** | **97.24** | **Dictionary validation costs ~24 µs, about 26% of the round trip.** So the published QuickFIX numbers overstate its latency relative to libhft by roughly a quarter, purely from a configuration difference. For a fair comparison the arms must match: either all validate, or none do. **Recommended: turn validation OFF for the QuickFIX arms** (cpp, /n, /J) to match libhft, and state it. Note the trade — validation is what caught the `ExecType='2'` conformance defect, so the *conformance* role needs a separate validated run. Performance and conformance want opposite settings, and should be separate lanes rather than one compromised configuration. ⚠️ The no-dictionary run reports `status=not …` rather than `pass`; that must be resolved before the 68.22 µs figure is used. Without a dictionary QuickFIX cannot parse grouped messages, which the harness's line-321 comment already warns hangs `md`/`md_nos_er` — so "validation off" may not be viable for the market-data scenarios at all. --- ## Transport and Solarflare run — same box, same day Merged here from the former `RECORD_2026-08-11_scan.md`. It covered the same box on the same date and each document was labelled the other's companion. Full run of libhft's benches on SCAN after the repo split, including the Solarflare paths. Rig, traps and transport results: [`SCAN_BYPASS_RIG.md`](../SCAN_BYPASS_RIG.md). Conditions: cores 19/18 isolated, C1E capped, microcode `0xb000040`, gcc 11.5, `irqbalance` off. Everything serialised — L3 is one shared 50 MiB instance across all 20 cores, so a concurrent build corrupts a measurement without interrupting it. ### Results **Correctness: 70/70 pass.** Build clean, 133 binaries. #### Microbenches (cyc/op unless noted, core 19) | bench | this run | pre-split (2026-07-25) | Δ | |---|---:|---:|---:| | `ParseBuffer` | 362.25 ns | 356.71 | +1.6% | | `ParseExtract` | 364.81 ns | 357.16 | +2.1% | | `GenLogon` | 99.59 ns | 92.74 | **+7.4%** | | **`GenNewOrderSingle`** | **147.22 ns** | **110.85** | **+32.8%** | | `GenExecutionReport` | 140.45 ns | 134.11 | +4.7% | | `bench_roundtrip` loopback p50 | 16,351 ns | 15,499 | +5.5% | #### TCPDirect FIX NOS→ER over the wire — 3 reps ``` fix_nos_er,tcpdirect-zf,141,20000,3726,3797,3839,3881,4122,5594,3800.8 fix_nos_er,tcpdirect-zf,141,20000,3717,3795,3835,3877,4145,4434,3799.6 fix_nos_er,tcpdirect-zf,141,20000,3719,3793,3831,3870,4087,5501,3796.4 ``` p50 **3797 / 3795 / 3793**, mean spread **0.1%** — the most reproducible arm measured on this box. p99/p50 = 1.022. #### Checksum — the klittlepage remediation also fixed the kernel | scenario | old (`hadd`) | now (`sad_epu8`) | |---|---:|---:| | `checksum_*_stream` autovec | 665 | 798 | | `checksum_*_stream` avx2 | **5,533** | **1,134** | Replacing the all-rights-reserved `hadd` sequence with `_mm256_sad_epu8` cleared the licence exposure **and** made the kernel ~5× faster. Autovec still wins on stream, but the gap is now 1.4× rather than 8.3×. ### The `GenNewOrderSingle` slowdown is a correctness fix, not a regression `ins/op` rose **664 → 732**, so this is different code rather than tuning. Cause: commit `d826a51` *"exact (mantissa, power-of-ten scale) decimal WRITE"*. Verbatim from its message: > The write path was the only place a decimal still had to become a double. > Inbound has always been exact — `FixUtils::parse_decimal_scaled` — but > outbound had no scaled writer at all, so a value that arrived exact, or came > from an SBE mantissa which is exact by construction, had to round-trip through > binary floating point to reach the wire. > > For prices that are exactly representable that round trip happens to be > lossless. But "happens to be" is not a guarantee to put under a price… A test > pins the difference with 2^53+1, where the double path demonstrably loses the > value and the scaled path does not. Outbound prices now stay in integers. **That is worth 36 ns on a NOS encode**, and the trade was made deliberately with a test pinning the failure case. ⚠️ **But it does not explain everything.** `GenLogon` has no decimal field and still slowed 7.4%; `ParseBuffer` +1.6%. So there is a **general +2–7% on top of the NOS-specific +33%** that is unaccounted for. Candidates, not yet bisected: `99996e8` (compiler-aware primitive tuning), `8db98c4` (harden FIX paths), `1b0277a` (harden benchmark latency paths). **Bisecting that residual is the open question.** ### ✅ The tuner: was blocked by Onload, now fixed and run **Resolved.** `run()` in `tune.py` now captures subprocess output through `tempfile.TemporaryFile` instead of pipes. Files have no EOF semantics to wait on, so Onload's leaked daemon can hold a descriptor harmlessly. Full run completes; profile installed at `oss/cpp/tools/libhft-tune/profiles/scan-broadwell-lowlat/` (cpu_sig_hash `94535b5c460381e3`, `-march=broadwell`). Verified against a minimal reproduction of the leak — a child that exits immediately while leaving a daemon holding its stdout, exactly what `onload_cp_server` does: ``` PIPE (old behaviour): HUNG FILES (new behaviour): returned immediately ``` #### Picks, and how SCAN differs from BEELINK | knob | BEELINK (znver4) | SCAN (broadwell) | |---|---|---| | `CHECKSUM` | `VPADDB_MASKTAIL` | `VPADDB_MASKTAIL` | | `DECIMAL_FORMAT` | `CLEAN_ARMS` | `CLEAN_ARMS` | | `FIELD_SPLIT` | `COMPOSED` | `COMPOSED` | | `MEMCPY_TINY` / `MEDIUM` | `DESCEND` / `HEADTAIL` | `DESCEND` / `HEADTAIL` | | `TIMESTAMP` | `CURRENT` | `CURRENT` | | **`FIND_BYTE`** | `SWAR` | **`SSE2`** | | **`INT_FORMAT`** | `SSE2_CLANG_ASM` | **`SSE2`** | | **`INT_PARSE`** | `SWAR8` | **`MASK8_DELIM`** | | `STRUCT_IDX` | *(knob postdates that profile)* | `SEQ` | ⚠️ **jeaiii is not the fastest integer formatter on either box.** On SCAN, `sse2` = 19.936 cyc/op vs `jeaiii` = 27.491 — **27.5% faster**, CV 0.39% and 0.15% respectively, so this is not sampling noise. BEELINK independently picked an SSE2 variant too. jeaiii remains the compiled-in default (`LIBHFT_NO_JEAIII=0`, license-clean and portable); the per-box tuned profile overrides it. That is the two mechanisms composing as designed, not a conflict — but it does mean **a build without `LIBHFT_TUNED_DIR` leaves ~27% on the table in `int_format` on both machines.** Other notable decisions: - `parse_mode`: **SWITCH to unrolled**, 7.235 vs 11.313 cyc/op (−36%) - `checksum`: KEEP `vpaddb_masktail` (26.875) — nothing cleared the 3% margin - ⚠️ AVX-512 absent on Broadwell, so the tuner **could not** decide avx512-vs-autovec for checksum. That decision still requires running the tuner on AVX-512 hardware.
Original diagnosis (kept — it is the reason for the fix) #### The tuner could not run on this box since Onload was installed `make tune` hung for 15 minutes having consumed **zero CPU** (`TIME 00:00:00`, `%CPU 0.0`), with nothing logged after `tune.py` started. Diagnosis: blocked in `poll()` (syscall 7) holding **two pipe read-ends with no child process alive**. That is `subprocess.communicate()` waiting on pipes whose write-end is held by a leaked grandchild. Onload's control-plane server daemonises — `onload_cp_server --daemonise` (PID 112004) is running — inherits the pipe, and never closes it, so the parent waits forever. Supporting evidence: the tuner **succeeded on this box on 2026-07-25**, before Onload was installed. The chosen remedy was the file-capture rewrite above, applied at the single `run()` choke point through which all of `tune.py`'s subprocess use passes, so the fix is complete rather than per-call-site. A 900s per-command timeout (`LIBHFT_TUNE_RUN_TIMEOUT`) was added as a second line of defence for a genuinely hung child rather than a leaked descriptor.
### Not run — needs one install The full six-arm matrix (`private/ops/scripts/run-wire-matrix.sh`) needs binaries in the root-owned bench directory, which requires sudo: ```bash sudo install -m 755 ~/code/libhft/bench/cpp/fixbench/bin/ping_pong \ ~/code/libhft/bench/cpp/fixbench/bin/ping_pong_zf \ ~/code/libhft/oss/cpp/bin/bench_roundtrip \ /usr/local/libexec/libhft-bench/ ``` Both `ping_pong` variants are **built** (`make ping-pong BINDIR=bin`, `make ping-pong-zf BINDIR=bin ZF_ROOT=~/tcpdirect`) and only need installing. ~~`bench/cpp/fixbench/Makefile` hardcodes a WSL `BINDIR`~~ — **fixed**. The default now probes for `/mnt/c`, so WSL keeps its Windows-reachable output dir and every other host gets a local `./bin`. No `BINDIR=bin` override needed. Once installed, `run-wire-matrix.sh` produces the whole matrix and the decomposition `engine_cost = fix_rtt − ping_pong_rtt` per transport — now valid for TCPDirect too, since `ping_pong` was templated on the transport. ### Open 1. **Bisect the residual +2–7%** not explained by the decimal-write fix. 2. **Run the six-arm matrix** (needs the install above). 3. Verify `PlainTransport` is byte-for-byte the old `ping_pong` by re-measuring the kernel/Onload arms against the recorded figures — claimed, not yet proven. 4. Multi-session zf: TCPDirect supports it (`zf_muxer`, repeated `zftl_accept`); the limit is `FixSocketZf` owning one stack per session. Needs a stack/session split and a zf-aware `CoreAcceptor` path.