# libhft cross-machine benchmark record — 2026-07-12

Rigorous, reproducible cross-engine cross-machine latency benchmark. This record captures
**how the test was run** (machines, cores, transport), **which code/version** was tested,
**scenario conformance**, **build/tuning + compile options**, and the **synthetic results**
(p50 / p99 / p99.9). Driven by the turnkey harness so the per-engine knowledge is reused, not
re-derived.

## 0. Two lanes (host constraints)

- **Native ULL lane** (libhft, nexusfix, libtrading, quickfix_cpp, fix8): **bench3 client ↔
  bench2 server** (bench box + prod). Free to use bench3.
- **CrossFIX lane** (and the CrossFIX-vs-Philadelphia notebook): **must run bench1 ↔ bench2**
  — CrossFIX is **license-locked to bench1/06**; it cannot run on the bench3 bench box. Both
  endpoints are PRODUCTION → verify idle cores on BOTH before each run.

## 1. Machines & topology

| Role | Host | Use | Core (pinned) | NUMA |
|---|---|---|---|---|
| **client (measure)** | sitea-bench3 | dedicated bench box | 40 | node0 (even) |
| **server** | sitea-bench2 | **PRODUCTION** (live FX) | 50 | node0 (even) |

- Both: **Intel Xeon Gold 6548N** (Emerald Rapids), 64c / 2 sockets, no HT, isolcpus=12-63,
  tsc=reliable. NICs: Solarflare **sfc** ens1f0 on node0. **Onload 9.0.1.86**. gcc 11.5.
- **Cross-machine fabric:** 10.0.1.x (bench3 ens1f0 = .6, bench2 ens1f0 = .5). Onload,
  `EF_POLL_USEC=100000` (high = spin; bench2 has no `latency` profile so plain `onload`).
- **Core-idle verification (mandatory before every prod run):** bench2 core 50 confirmed 0.0%
  busy over ≥2 s AND no prod-app affinity includes it (only per-CPU kernel threads
  migration/50, ksoftirqd/50, kworker/50 — dormant). Prod app runs on low cores 1–11; the
  high cores 26–63 are affinity-forbidden for prod → structurally free.

## 2. Harness / how the test is run

- Tool: `bench/cpp/fixbench/tools/open_source_fix_engines/open_source_fix_engine_native_stage5.py`
  (+ `_go_/_dotnet_/_dfx_/_rust_` variants for the non-native engines). Captures
  mean/min/p50/p90/**p99/p999**/p9999/max per engine/scenario; deploys + drives client/server
  cross-machine; encodes each engine's distinct CLI (why the manual per-engine approach failed
  — e.g. nexusfix rejects libhft's `--server-core`).
- Invocation shape (reuse prebuilt binaries, **no rebuild on the prod box**):
  ```
  python3 open_source_fix_engine_native_stage5.py \
    --mode remote --transport onload --skip-compile \
    --engines libhft,nexusfix,libtrading,quickfix_cpp,fix8 \
    --scenarios nos_er,md,md_nos_er,tick_to_trade \
    --client-host sitea-bench3 --client-core 40 \
    --server-host sitea-bench2 --server-core 50 --server-address 10.0.1.5 \
    --remote-dir ~/libhft_open_source_fix_engines
  ```
- SSH: multiplexed (ControlMaster) to avoid sshd throttling during the many harness
  connections.

### 2.1 Threading / cores-per-side (important)

The stage-5 contract is **single-threaded closed-loop ping-pong** (`bench_roundtrip.cpp`: the
`--stage5` path is "closed-loop ping-pong path only, rate ≤ 0"; every engine's
`*_stage5_xmachine` implements the same one-thread request→reply loop). The harness pins each
side with **`taskset -c <single core>`** — so the comparison is deliberately **1 thread / 1
core per side**, which is the fair apples-to-apples footing (measures each engine's per-thread
round-trip efficiency). Server: one server thread on `--server-core`. Client: one thread on
`--client-core`.

**A second, complementary regime exists but is NOT in the cross-engine table:** libhft's bench
also has a **the internal harness split mode** (`run_split_client`, rate-driven) that uses **3 client cores**
(measure + worker + poller) and includes the app↔engine thread hand-off — a deployment-realistic
number. Only libhft's bench supports it, so it's run for libhft *separately*, not as a
cross-engine row. This is also why the plain-`nos_er` ping-pong reads ~6.7 µs while the stage-5
xmachine ping-pong reads ~8.8 µs (heavier message contract, not a threading difference).

## 3. Code version under test

- Repo HEAD: **37a9232** (branch iv-front-vol-research). This session's libhft changes are all
  cold control-plane (W1-Java audit, W3 runtime dictionary, W5 FIX50SP2, W7 audit-replay CLI) —
  the hot path is byte-identical to pre-session (verified: bench3 A/B HEAD vs pre-W5 = -0.66%
  nos_er, within noise).
- Engine binaries: the on-box `~/libhft_open_source_fix_engines/stage5/build_native_direct/
  *_stage5_xmachine` set (libhft, nexusfix, libtrading, quickfix_cpp, fix8). libhft's is
  `bench_roundtrip`. **TODO: rebuild libhft's with the TUNED library (§4) and record the SHA.**

## 4. Build / tuning + compile options  ⚠️ IN PROGRESS

Per requirement: libhft (cpp/java/.net) must use the **locally-tuned** libraries, and the tuned
+ compile options must be recorded.

- **Compile options (base CXXFLAGS):**
  `-O3 -flto=auto -march=native -mtune=native -falign-functions=16 -falign-jumps=16 -std=c++11`
  `-fno-plt -fno-semantic-interposition -fno-stack-protector -DUSEBUFFER`.
- **Tuning flow:** `make-tune` (tools/libhft-tune) runs box-local kernel-selection microbenches
  → `tools/libhft-tune/out/{libhft_tuned.h, libhft_tuned.mk}`. Consumed via
  `make LIBHFT_TUNED_DIR=<out> bench_roundtrip` which adds `LIBHFT_TUNED_CXXFLAGS` +
  `-DLIBHFT_TUNED_BUILD=1 -include libhft_tuned.h`. On Emerald Rapids (bench3) tuning is expected
  to select the psadbw/SWAR kernels (per prior [[project_libhft_autotune]] campaign: EMR→psadbw_ma).
- **libhft-java:** the JNI `.so` is built from the same C++ headers → same tuned kernels; rebuild
  the `.so` with the tuned dir.
- **libhft-net:** separate managed engine; ServerGC=true / ConcurrentGC=false (from its csproj).
  No cross-machine harness yet → NOT in this cross-machine table (in-process codec only).
- **Tuned kernel selections (bench3 / Emerald Rapids, `make tune` 2026-07-12)** — from the
  tune self-test:
  `checksum=psadbw_ma · parse=loop · find_byte=swar · memcpy_tiny=overlap · timestamp=current ·
  int_format=jeaiii · int_parse=swar8` (structural-index impl swept 0/1/2). These are the
  box-optimal kernels the tuned build bakes in via `libhft_tuned.h`.
- **Tune run (bench3 core 58, 2026-07-12):** per-decision result — where candidates fall
  inside the 2% CV noise gate the tune KEEPS the robust default (checksum psadbw_ma, find_byte
  swar, int_format jeaiii, int_parse swar8); where clearly better it SWITCHES (10-level MD:
  `avx2 195 vs seq 1286 cyc/op` → avx2). The "RUN QUALITY INCONCLUSIVE" banner is conservative
  (turbo enabled on bench3 makes the AVX-512-downclock decision frequency-policy-dependent — the
  tool recommends the AVX-512 call be made on the deploy box under its prod governor). Net: the
  prescription (`out/libhft-tune-result.json`) is usable = tuned-where-confident + safe-defaults.
- **TODO CLOSED (2026-07-20): tuned-vs-untuned A/B ran end-to-end on bench3 — VERDICT: DO
  NOT deploy the prescription globally.** Fresh `make tune TUNE_CORE=54` (isolated core;
  turbo-policy banner remains, as documented) then `make LIBHFT_TUNED_DIR=tools/libhft-tune/out
  BINDIR=bin-tuned … bench_components bench_roundtrip`, interleaved 3× vs the untuned build:
  - loopback nos_er RTT p50: 5497 → 5416 ns (**−1.5%**, real but modest);
  - components: MD-decode fixtures genuinely faster (compact MD parse −6.8/−9.3%, observed
    decode flows −3..−5%) **but 70 of 185 ops ≥10ns regress >+3%** (mean +2.28%;
    `field_parse_short` +59%, parse/logon flows +8–12%) — far beyond the ±0.0%-median /
    4-outlier band of a pure rebuild A/B, i.e. systematic.
  - Root cause: the tuner scores kernels in ISOLATION (cyc/op in its own harness); the
    `FIX_FIELD_TAG_PARSE_MODE_UNROLLED` / `FX_FIND_BYTE_MEMCHR` picks lose IN CONTEXT
    (I-cache/inlining) on many small-parse paths while the MD/memcpy picks win.
  - Follow-up direction: give the tuner an in-context gate (compile bench_components with the
    candidate prescription and require no >+3% regression on the hot fixture set before a pick
    is emitted), or cherry-pick per-kernel from the A/B table above.
  - **In-context gate BUILT + RUN (2026-07-20, `tools/libhft-tune/incontext_gate.py`): FINAL
    VERDICT — nothing in the prescription is deployable.** Methodology: 13 build variants
    (base / full / flags-only / picks-only / metadata-only "noop" / each of 8 picks singly /
    gated combo), interleaved 3x on an isolated core, noise-aware grading (a REAL per-op effect
    must escape the reference build's run-to-run range AND clear max(3%, 2.5ns); picks graded
    against the noop reference so the common LIBHFT_TUNED_BUILD effect cancels). Findings:
    (a) `FX_FIND_BYTE_IMPL=MEMCHR` is catastrophically bad in context (+59% field parse,
    +39% delim-scan stream, 80 real regressions) — the tuner's isolated score does not
    transfer; (b) `FIX_FIELD_TAG_PARSE_MODE=UNROLLED` is double-edged (53 real wins incl. MD
    decode −12%, 7 real regressions +13% on direct-decode) — a per-workload trade, never a
    global default; (c) every other pick is indistinguishable from build-layout noise, and the
    striking "−22% scalar checksum/finalise stream" readings that appear in several variants
    are LAYOUT BISTABILITY (two stable loop layouts, 217 vs 168 ns, flipping per build — absent
    from noop-vs-base and NOT reproduced by the gated combo). The gate's final combination
    check exists precisely for this and refused the deploy. The engine's hand-chosen defaults
    stand; any future prescription must pass `incontext_gate.py` before reaching a build.

## 4.1 Box config analysis + target-core selection (2026-07-12)

Full read-only profile of bench1/06/07 (drove the tune fix + the bench core choice):
- **All three:** `isolcpus=12-63`, **governor=performance, turbo ENABLED (no_turbo=0)**, **no
  `nohz_full`/`rcu_nocbs`** → every core still takes the periodic timer tick. NIC (`ens1f0`) on
  **NUMA node0 = even cores** → bench/tune cores must be even (node0) for NIC locality.
- **Per-core device-IRQ pattern:** a low uniform baseline (~1.1M lifetime ≈ <1 IRQ/s) on most
  cores, but a handful **own the active Solarflare RX queues** and take millions–billions
  (bench3: 45, 47, 31, 35, 41, 43; bench2: 26, 47, 31, 43, 27, 28). **Avoid those.**
- **No perfectly-quiet core exists** (no nohz_full + IRQs spread) → a pure mean/stdev CV will
  always be borderline on µs microbenches → the tune tool must be robust (see §4.2).
- **Core selection is now automated** — `tools/pick_bench_core.py` (stdlib, runs on-box):
  picks isolated + NUMA-local (`--nic ens1f0`→node0) + idle + **low device-IRQ** cores, with a
  `--cores 26-63` range to stay in a prod box's app-forbidden high range. Validated: on bench2
  it correctly **skips core 26** (1.3B IRQs = the top NIC-RX queue) and returns 28; on bench3
  returns 26. The bench launch and the tune share this policy (tune's `find_free_isolated_core`
  is now the same idle+IRQ-aware logic).
- **Chosen cores (auto-picked, verified):** bench server bench2 = **28**, client bench3 = **26**
  (both node0, idle=1.0, 0 IRQ/s); tune → any clean isolated core the tool selects. Still
  re-verify idle immediately before each prod run.

## 4.2 Auto-tune fix (robust variance gate)

`tune.py` decided kernels on the **median** (robust) but gated confidence on a **mean/stdev CV**
(outlier-sensitive) — so one tick/IRQ-perturbed rep (unavoidable per §4.1) tripped a FALSE
`INCONCLUSIVE` and discarded a decision the median already made. **Fix:** `cv_of` → robust
**scaled-MAD/median** CV, and `adaptive_samples` convergence → median/MAD half-width. Now the
gate agrees with the decision (still flags genuine bimodal dispersion; no longer over-fires on
interrupt noise). Re-run on core 46 to confirm the `recommend_rerun` banner clears.

## 5. Scenario conformance + bench-spec compliance audit (2026-07-12)

The stage-5 xmachine scenarios (`nos_er`, `md`, `md_nos_er`, `tick_to_trade`) are a fixed
message contract every engine must reproduce byte-correctly (the harness fails a row that
doesn't match the expected ER/MD reply). All listed engines re-confirmed `stage5-pass` on this
run (native lane) / `pass` on the Java lane.

Audited the three axes the comparison hinges on — **MD depth, timestamp measurement point, and
message/work parity** — to confirm every engine measures the *same thing*:

**(a) MD depth = 10 levels** (contract §: `md.levels=10` ⇒ request `264=10`, snapshot `35=W`,
`268=20` bid/ask entries; tick_to_trade top-of-book `268=2`). Verified in *both* lanes on the
fresh runs:
- Native lane (client logs): `264=10`, `268=20`, `mdLevels=10`, `--md-levels 10 --ttt-md-levels 1`.
- Java lane (`*.audit.json`): `md_levels_arg=10`, `md_levels_effective=10` for **all four** engines
  (libhft-tcp, crossfix-socket, quickfix-j, philadelphia).

**(b) Timestamp measurement point — identical logical boundary in both lanes:** `t0` at task
start (send), `t1` at receive, client-side wire-to-wire RTT (contract §: "`t0` is stamped at the
actual task start, not the intended send time"; measured window excludes warmup; a ResendRequest
inside the window invalidates the row).
- Native (`bench_roundtrip.cpp`): `t0` stamped on the `35=V`/`35=D` send, `t1` on the final
  `35=W`/`35=8` receive (`sample_fix_client_to_server_t0` … `sample_fix_server_to_client_t1`).
  `tick_to_trade` correctly moves t0 to the **server/feed side** (`sample_fix_server_to_client_t0`
  = 35=W tick), matching "server-side tick-to-trade measured on the feed side".
- Java (`LatencySink.end_to_end`): `sample(System.nanoTime() − startNanos)` — same send→receive
  span. Same definition, different implementation (the two lanes are reported separately in the
  notebooks; they are not merged into one table).

**(c) Work parity — cross-language extraction sink.** Each engine runs the same extraction work
(`sink = sink*31 + value` over the parsed business fields; anchor `EXTRACTION_SINK_GOLDEN =
8941056108728227747` for the ER example) so a fast-but-cheating parser can't win by skipping
fields. Per-scenario the sink is deterministic and cross-engine-identical (libhft `md`
sink=`14762292562046071932`; client-side `nos_er` sink=0 by design — the *client sends* the order,
the *server* extracts it). The harness `stage5-pass` gate is what enforces this per row; the
latency logs only echo the sink for libhft.

**Verdict: compliant.** All engines in both lanes use 10-level `md`, the same send→receive RTT
boundary, and the same message/extraction contract — the p50/p99/p99.9 tables below are
apples-to-apples within each lane.

## 6. Results — p50 / p99 / p99.9 (µs), cross-machine onload

⚠️ PRELIMINARY — only libhft measured so far (untuned build), pending the tuned rebuild + the
full harness batch.

**libhft tuned vs untuned — measured (plain ping-pong, cores server bench2:28 / client bench3:26,
onload, 5 reps each):**

| Engine | Build | mode | nos_er p50 | nos_er p99 | nos_er p99.9 |
|---|---|---|---:|---:|---:|
| libhft | **tuned** (SPR march + psadbw_ma/swar/unrolled/overlap) | ping-pong | 6.70 | 7.05 | 7.90 |
| libhft | untuned | ping-pong | 6.70 | 7.03 | 7.71 |

**Finding: tuned ≈ untuned on cross-machine RTT** — the onload cross-machine path is
**transit-dominated** (NIC + wire + onload stack), so the tuned COMPUTE kernels (parse/checksum/
find_byte) don't move the network-bound round-trip. The tuning wins live in the compute
microbenches (bench_components: encode/parse cyc/op), not the cross-machine RTT. (Consistent with
the 2026-06-28 campaign: onloaded RTT ≈ −3% while compute microbenches −15%.)

**Parsing-only microbenches (bench3 core 30, in-process, fresh 2026-07-12) — where tuning DOES
show (compute, not transit):** tuned vs untuned p50 ns — parse_md_incremental 876 vs 1007
(−13%), parse_md_request 142 vs 154 (−7.8%), parse_execution_report 151 vs 161 (−6.2%),
parse_logon 110 vs 116 (−5.2%), parse_new_order 138 vs 144 (−4.2%), field_parse_short 17 vs 17.
Confirms the split: **tuning wins the parse/compute path (−4…−13%), the onload cross-machine RTT
is transit-bound so it doesn't move.**

**Cross-engine comparison — ALL FRESH this session (no old data), both transports (native-TCP
`plain` + `onload`), medium profile (10k samples × 3 runs), tuned libhft, auto-picked cores
(server bench2:28 / client bench3:26). Native lane RUNNING; Java/CrossFIX lane (bench1↔06)
queued. Prior-campaign numbers below are SUPERSEDED and retained only until the fresh table lands.**

### FRESH native lane — onload, stage-5, 10k samples × 3 runs, tuned libhft, cores 26/28 (2026-07-12)

Median of 3 runs, p50 / p99 / p99.9 µs. Raw `.bin` distributions collected to
`/mnt/c/Temp/libhft/fresh-bench-2026-07-12/native-onload/` (for the notebook CDF plots);
summary at `.../summary/fresh_onload_summary.csv`.

| Engine | nos_er | md | md_nos_er |
|---|---|---|---|
| **libhft** (tuned) | 8.85 / 10.16 / 11.0 | 10.20 / 11.13 / 12.0 | 19.13 / 20.64 / 22.1 |
| libtrading | 9.08 / 10.21 / 11.25 | 9.07 / 10.22 / 11.51 | 18.01 / 19.79 / 21.2 |
| nexusfix | 14.61 / 15.41 / 16.28 | 17.36 / 18.19 / 19.0 | 33.21 / 34.19 / 35.3 |
| fix8 | 30.4 / 31.9 / — | 84.3 / 86.2 / — | 116.3 / 118.8 / — |
| quickfix_cpp | 121.8 / 123.9 / 124.9 | *dep-fail* | *dep-fail* |

libhft leads the ULL peers (1.0× vs libtrading 1.03×, nexusfix 1.65× on nos_er). Fresh libhft
nos_er 8.85µs matches the prior 8.80µs (validates).

### FRESH native lane — PLAIN (native-TCP), same config (2026-07-12), p50 µs

| Engine | nos_er | md | md_nos_er | onload nos_er | onload benefit |
|---|---|---|---|---|---|
| **libhft** (tuned) | 12.63 | 14.14 | 26.97 | 8.85 | **−30%** |
| libtrading | 14.47 | 13.67 | 28.71 | 9.08 | −37% |
| nexusfix | 14.52 | 16.36 | 32.07 | 14.61 | ~0% |
| quickfix_cpp | 125.8 | — | — | 121.8 | ~0% |

**Finding: onload (kernel-bypass) helps the ULL engines hugely (libhft −30%, libtrading −37%)
but NOT the heavier ones** (nexusfix, quickfix_cpp ~unchanged) — kernel-bypass removes the
syscall/transit cost, which only dominates when the engine's compute is already tiny. libhft's
onload advantage over nexusfix widens to 1.65× because libhft actually exploits the bypass. Both
transports' raw `.bin` distributions are under `/mnt/c/Temp/libhft/fresh-bench-2026-07-12/
{native-onload,native-plain}/`; summaries in `.../summary/`.

**quickfix_cpp** md/md_nos_er and **the Java/CrossFIX lane** remain — see §9.

## 9. Status — COMPLETE (2026-07-12)

**Both performance report notebooks regenerated from ALL-FRESH data (commit 8dfee1e):**
- `fix_engine_latency_distributions.ipynb` — native lane (libhft-cpp tuned / nexusfix /
  libtrading, both transports) + Java lane (libhft-java / crossfix / quickfixj / philadelphia,
  both transports); curves=15055.
- `crossfix_philadelphia_comparison.ipynb` — fresh CrossFIX-vs-Philadelphia.
Builders (`build_*_notebook.py`) updated to point at the fresh run dirs (historical dirs
dropped). Prod hygiene: tuned-binary swap reverted; bench1/06/07 verified clean, bench cores idle.

### Original remaining items — resolved

- **DONE fresh:** native onload lane (libhft-tuned, nexusfix, libtrading, fix8; quickfix_cpp
  nos_er) + raw `.bin` collected/organised + parsing microbenches (tuned vs untuned).
- **RUNNING:** native **plain** (native-TCP) lane (both-transports requirement).
- **REMAINING (large):**
  1. **quickfix_cpp** md/md_nos_er fail cross-machine (runtime .so dep on the client box) — needs
     its libquickfix deployed to bench3 or a static build.
  2. **Java / CrossFIX lane — RUNNING (2026-07-12).** FixBench2 `run_xmachine_matrix` via
     `run_sitea_bench12_smoke.py`, server **bench1** → client **bench2** (CrossFIX license OK on
     both; jars + `libhftbench.so` deployed). Engines qf/ph/lh/cf, scenarios nos_er/md/md_nos_er,
     **both transports** (plain+onload), profile medium (the internal harness adaptive), Java 21. **Same rigorous
     core selection** (`pick_bench_core.py`, per-box): server bench1 cores **26,28**; client
     bench2 **32/34/36** (measure/worker/poller — the picker avoided bench2's noisy core 26). All
     verified idle. Output → `/mnt/c/Temp/libhft/fixbench-runs/<ts>_sitea_bench12_fixbench_smoke/`.
     ~40 min (24 rows × adaptive). Smoke validated end-to-end (crossfix rc=0).
  3. **Regenerate both notebooks** from the fresh raw samples (native `.bin` in place; Java lane
     pending) via `scripts/build_*_notebook.py`.
- **Prod hygiene TODO:** restore `libhft_stage5_xmachine` from `.orig` on bench2/07 after the
  plain run; verify no orphans + cores idle.

**PRIOR campaign numbers (stage-5, 1 core/side — SUPERSEDED by the fresh table above; retained
session; see §6.1):** libhft 8.80 / 10.18, libtrading 9.40 / 10.71, NexusFIX 14.79 / 15.62,
QuickFIX-C++ 161.8 / 164.9, QuickFIX/N ~1084 (p50/p99 µs). NOTE the mode gap: libhft's fresh
number above is **plain ping-pong (6.70µs)**; the stage-5 xmachine contract reads ~8.80µs — so
the cross-engine row for libhft is 8.80, not 6.70. A full FRESH cross-engine batch (harness,
stage-5, all engines, tuned libhft) is the remaining execution step.

## 7. Reporting — the 2 performance report notebooks (objective)

The campaign closes into two repo notebooks (regenerated from the benchmark raw-sample
artifacts by their builder scripts — the scripts render, they do not launch benches):
1. **`bench/java/fixbench/notebooks/fix_engine_latency_distributions.ipynb`** —
   builder `scripts/build_fix_engine_latency_distribution_notebook.py`; consumes raw-sample
   CSVs under `/mnt/c/Temp/libhft/{open-source-fix-engines,fixbench-runs}`. Cross-engine
   latency distributions (p50…p9999, histograms).
2. **`bench/java/fixbench/notebooks/crossfix_philadelphia_comparison.ipynb`** —
   builder `scripts/build_crossfix_philadelphia_notebook.py`; CrossFIX vs the Philadelphia
   version used by the Java-8 risk stack.
**Pipeline:** tune → build tuned libhft → harness batch (writes raw-sample CSVs to the
workspace, staged under `/mnt/c/Temp/libhft/…`) → re-run both builder scripts → **review** the
rendered notebooks for correctness. TODO once the batch data lands.

## 8. Prod-hygiene lessons (this campaign)

- **SSH throttling:** rapid sequential ssh triggers sshd rate-limiting (exit 255, no output).
  Fix: ControlMaster multiplexing (added to ~/.ssh/config for the sitea hosts).
- **Orphaned servers:** the bench server keeps busy-spinning AFTER the client finishes — it must
  be explicitly killed. `comm` truncates to 15 chars, so `pkill -x <binary>_stage5_xmachine`
  misses it; kill by PID or `pkill -9 -f <binary>`. ALWAYS verify after: no residual proc +
  target core ~0% + /tmp clean. (Caught + cleaned a spinning libhft server on core 50 this run.)
- Per-engine CLIs differ → use the harness, don't hand-roll invocations.

## 10. ALIGNED-HARNESS re-run — shared cpp/hft-bench, bench1↔06 (2026-07-12, later)

Supersedes §6 for cross-engine comparison. The prior §6 lane ran **closed-loop `--stage5`**
on bench3↔06; that made libhft-cpp look *slower* than libhft-java (which ran paced) — a
methodology artifact, not a real gap. This re-run routes **every** engine through the new
shared `cpp/hft-bench` harness so all lanes (C++ / Java / C#) measure identically.

**What changed (the alignment):** paced load at `--throughput 10000/s` (== Java default),
**internal** multi-run (`--runs 3 --adaptive-runs --max-runs 6`) with the run-to-run CV
**stability gate**, mean-of-per-run percentile aggregation, CO-free service-time (t0 = actual
issue, scheduler-slip a separate probe), and **serve-until-client-disconnect** servers (lean
hot loop). Machine pair = **bench1 (server) ↔ bench2 (client)** — the *same pair as the Java
lane*, so the earlier machine-pair confound is gone. Profile `aligned` (warmup 50 000 /
iters 100 000), external runs=1 (multi-run is internal → each engine invoked once). Cores
picked by `pick_bench_core.py` (isolated, NUMA0-local, idle, low-IRQ) = core 12 both boxes.
Each row = 300 000 samples (3 internal runs). Raw `.bin` under
`/mnt/c/Temp/libhft/open-source-fix-engines/stage5/*_aligned_*_remote/run_1/`.

### p50 / p99 / p99.9 µs — PLAIN (native-TCP)

| Engine | nos_er | md | md_nos_er |
|---|---|---|---|
| **libhft** | **12.7 / 13.9 / 16.7** | **13.8 / 14.9 / 18.1** | **25.7 / 27.5 / 30.3** |
| nexusfix | 17.6 / 18.8 / 21.7 | 19.9 / 21.2 / 24.4 | 36.3 / 37.7 / 41.4 |
| libtrading | 16.1 / 18.8 / 20.8 | 19.7 / 22.4 / 24.6² | 34.6 / 38.6 / 40.5² |

### p50 / p99 / p99.9 µs — ONLOAD (busy-poll `EF_POLL_USEC=100000`, cores 40/40)

Corrected: re-run with onload busy-poll (see the paradox section below). Supersedes the earlier
interrupt-driven onload numbers, which were ~2–3µs high across the board.

| Engine | nos_er | md | md_nos_er |
|---|---|---|---|
| **libhft** | **6.9 / 7.4 / 8.3** | **8.0 / 8.5 / 9.7** | **15.1 / 15.8 / 17.2** |
| nexusfix | 8.0 / 8.4 / 9.0 | 10.6 / 11.3 / 11.8 | 19.7 / 20.5 / 21.0 |
| libtrading | 8.7 / 9.6 / 10.8 | 12.5 / 13.4 / 14.7² | 21.3 / 22.6 / 23.8² |

vs **libhft-java** onload (aligned, same pair): nos_er 7.3 · md 8.3 · md_nos_er 16.0 — **libhft-cpp
is faster on all three** (6.9/8.0/15.1), as physics requires (java is JNI over the same core).

**Findings.** libhft is fastest on **every** scenario/transport (aligned, apples-to-apples).
libtrading nos_er is neck-and-neck (onload 9.4 vs libhft 9.2 — RTT-domination noise per the
libtrading speed investigation, not a technique win), **but on the MD-heavy path libhft clearly
leads** (md 13.8 vs 19.7 plain; 10.5 vs 13.2 onload) — the aligned measurement confirms the
investigation's thesis: libtrading's eager per-field `strtod` parse costs it on 10-level `md`,
where the earlier un-aligned run had shown it "edging" libhft (RTT noise). Onload helps the ULL
engines most (libhft nos_er −28%, libtrading −42% vs plain; nexusfix −15%). The libhft-java <
libhft-cpp paradox is resolved by construction — one harness, one methodology, one machine pair.

### Engine caveats (environment/engine, NOT the harness)
1. **libtrading md/md_nos_er** (² above): the summary line `double free`d because libtrading's
   48-field receive parser can't hold a conformant 10-level snapshot (63 fields). The crash is in
   *summarize*, AFTER measurement, so the raw `.bin` are **valid** (per-run p50 stable to ±0.1µs)
   — the ² cells are computed from them (mean-of-per-run). For a clean end-to-end run, rebuild
   `libtrading.a` with `FIX_MAX_FIELD_NUMBER≥64`.
2. **fix8 — link fails on bench1**: bench1 glibc too old (`@GLIBC_2.38`/`@GLIBCXX_3.4.32`) for
   its prebuilt libs (builds fine on bench2). Box-toolchain, not the adapter.
3. **quickfix_cpp md — client hangs** (nos_er fine); md-specific; was already dep-failing.

### Cross-language parity
All three harnesses now report the same evidence: **C++** (`cpp/hft-bench`, this run),
**Java** (`java/hft-bench`, the reference), **C#** (`csharp/hft-bench` — multi-run + mean-of-
per-run `AggregateLatencyStats` + `ProbeStabilityStats` CV gate + adaptive + `ceil(p·n)-1`
percentiles + byte-identical `HFTBRW1` raw). C#'s `.bin` feed the same notebook CDF builders.

### The libhft-java < libhft-cpp paradox — DEFINITIVELY resolved (two config asymmetries)

The original paradox (JNI libhft-java appearing FASTER than native libhft-cpp — impossible, since
java is JNI over the *same* C++ core) was **two** methodology asymmetries, now both closed:

| Transport | Asymmetry | Before | After (aligned) |
|---|---|---|---|
| **plain** | load model: native ran **closed-loop** `--stage5`, Java ran **paced** | cpp 8.85 "slower" | cpp 12.7 **<** java 15.0 ✓ |
| **onload** | native `onload` was **interrupt-driven**; Java used `EF_POLL_USEC=100000` **busy-poll** | cpp 9.2 **>** java 7.3 ✗ | cpp **6.9** **<** java 7.3 ✓ |

Controlled re-run (bench1:12 ↔ bench2:16, `EF_POLL_USEC=100000`, aligned knobs, 150k samples,
stability=PASS): **libhft-cpp onload nos_er = 6.925µs, faster than libhft-java's 7.3µs** — as
physics demands (cpp ≤ java at equal footing). The `env EF_POLL_USEC=100000` prefix is now
baked into the launcher's `transport_prefix` (`--onload-poll-usec`, default 100000). The §10
ONLOAD table above **has been re-run WITH busy-poll** (cores 40/40): every engine dropped ~2–3µs
(libhft nos_er 9.2→6.9, nexusfix 14.9→8.0, libtrading 9.4→8.7), libhft still leads, and
libhft-cpp is now faster than JNI libhft-java on **all** onload scenarios (6.9<7.3, 8.0<8.3,
15.1<16.0). Net: with both asymmetries removed, **libhft-cpp is fastest on both transports** and
the paradox is gone by construction.

## Three-way engine comparison — cpp vs java vs .net (2026-07-21, bench3)

First like-for-like same-box comparison of all three libhft implementations. One protocol:
closed-loop NOS→ER loopback over real TCP, FIX 4.4 session, client-owned RTT stamps, 50k
warmup + 100k measured, **interleaved 3 rounds** (cpp→java→net per round) on the identical
core layout (aux 52 / client-measure 54 / server 56, isolated). Lanes: cpp =
`bench_roundtrip --role loopback`; java = `tools/validation/LoopbackBenchJava.java`
(java-driven `sendNow` + `receive()` — the fastest java path, same two-thread topology);
.net = `tools/loopback-bench` (`ManagedSession` BusySpin, ping-pong driven from the owner
thread handler, engine-pinned owners; needs `DOTNET_PROCESSOR_COUNT=64` — the runtime's
ProcessorCount underestimates under restricted affinity and the pin guard rejects high cores).

| lane | p50 (3 runs, ns) | p99 | p999 | max |
|---|---|---|---|---|
| **libhft-cpp** | **5535 / 5552 / 5557** | ~6030 | ~6900 | 10–15µs |
| **libhft-java** | 6023 / 6096 / 6098 | ~6500 | ~7200 | ~30µs |
| **libhft-net** | 5996 / 6100 / 6109 | ~6560 | ~7200 | **10–12µs** |

Readings:
- **cpp leads at 5.55µs**; **java and .net are statistically TIED at ~6.1µs** (+9.9% / +9.8%
  over cpp — java's ~0.55µs = the documented JNI + event-ring crossing cost; .net matching it
  is a strong result for a fully managed independent engine).
- **Tails invert the stereotype**: .net's max (10–12µs, ServerGC/no-concurrent, zero-alloc hot
  path) is TIGHTER than java's (~30µs JVM safepoint/background blips); both p999s ~7.2µs.
- Same-harness-model caveat: each lane stamps in its own harness (identical closed-loop shape,
  so cross-lane offsets are far tighter than the t2t cross-harness case, but sub-100ns deltas
  should not be ranked).


---

## TCPDirect (Solarflare zf, native ULL-TCP) vs onload vs plain — the wire floor (2026-07-21)

**Question:** how much latency is left on the table below onload? Onload emulates POSIX
sockets over kernel-bypass; TCPDirect (`libonload_zf`, the `zf`/zocket API) is Solarflare's
*native* kernel-bypass TCP — no sockets-API emulation layer. This measures the raw TCP-echo
RTT floor each transport gives on the same wire, so a future `FixSocketZf` transport's ceiling
is known.

**Method — apples-to-apples, same path, same payload, full round-trip:**
- Path: **bench3 (client/measure) ↔ bench1 (server), Solarflare `ens1f0` 10.0.1.6↔.4**,
  same SF subnet/switch fabric as the bench1↔06 FIX benches. bench1 = fully-idle prod box;
  server side is a passive responder on an **unused port**, pinned to an **idle isolated core
  (50)**, transient single VI — no more invasive than the onload FIX runs. Client on the bench
  box (bench3, core 40). Cores re-verified 0.0% busy before the run; 0 bench procs after.
- Payload **128 B**, full ping→pong→ping round-trip, busy-spin both ends, 50k warmup + 100k
  measured.
- plain/onload via **`ping_pong`** (single-clock rdtscp, `--slots 16`=128B); onload =
  `EF_POLL_USEC=100000 onload` busy-poll (bench1 lacks the `latency.opf` profile — use the env
  var, which sets the one knob that matters). TCPDirect via **`zftcppingpong -s 128 -f`** (the
  `-f` table is the *true* full-RTT; the "mean round-trip time" headline is `s.mean*2` =
  double-counted when `-f` is set — a tool bug, ignore it).

| transport | min | **p50** | p90 | p99 | p999 | max | p99−p50 |
|---|---|---|---|---|---|---|---|
| plain (kernel TCP) | 11.69 | **12.68** | 12.84 | 13.47 | 15.15 | 34.1 | 0.79 µs |
| onload (busy-poll) | 6.50 | **6.66** | 7.02 | 7.12 | 7.67 | 11.5 | 0.46 µs |
| **TCPDirect (zf)** | 5.31 | **5.45** | — | 5.55 | — | 10.4 | **0.10 µs** |

(µs; TCPDirect stddev **52 ns** over 100k.)

**Readings:**
- **onload −47% vs plain** (12.68→6.66µs); **TCPDirect −18% vs onload** (6.66→5.45µs),
  **−57% vs plain**. The ~1.2µs onload→TCPDirect gap matches Solarflare's published delta
  (dropping the sockets-emulation layer).
- **TCPDirect is in a different determinism class:** p99 is only **95 ns** above p50 and its
  *p99 (5.55µs) beats onload's p50 (6.66µs)* — even TCPDirect's tail is faster than onload's
  median. stddev 52ns vs onload's ~0.2µs, plain's jittery 34µs max.
- **What it means for libhft:** the onload FIX nos_er RTT (~9.2µs on this fabric) carries
  ~2.5µs of FIX parse/build over the 6.66µs onload wire. Re-hosting the same engine on a
  `FixSocketZf` (zf) transport would target ≈ 5.45 + 2.5 ≈ **~8µs FIX RTT** — the real prize,
  now unblocked (TCPDirect built from source on bench3, tag tcpdirect-9.0.1.31 vs onload
  9.0.1.86, no sudo/no AMD account: `~/tcpdirect/build/gnu_x86_64/{lib/libonload_zf.so,
  bin/zf_apps/…}`, headers `~/tcpdirect/src/include/zf/`).

Caveat: plain/onload measured with `ping_pong`, TCPDirect with `zftcppingpong` — different
tools, but identical workload (128B TCP echo, rdtscp full-RTT, busy-spin), and the ~1.2µs
gap dwarfs any timing-harness delta. Single 100k run each (distributions tight enough that
interleaving wasn't needed for the qualitative conclusion).

---

## FIX-over-TCPDirect: FixSocketZf transport + the FIX-frame RTT (2026-07-21)

The wire-floor section above measured *raw byte echo* over zf. This re-hosts the actual
libhft engine on zf: **`FixSocketZf`** (include/FixSocketZf.h) is a duck-typed sibling of
FixSocket/FixSocketSsl that runs the FIX engine over Solarflare's native TCPDirect (`zf`)
stack -- `CoreSession<FixSocketZf>` / `CoreInitiator<FixSocketZf,App>` compile and run
unchanged. v1 covers the initiator + a single-session server (the acceptor's fd-adoption
model doesn't map to zf's whole-zocket accept -- see the header note; multi-session is a
follow-up). RX is copy-based via `zft_recv` (reactor pumped per `ReadMsg`); TX via
`zft_send_single`.

**Bench (`make zf` -> bench_zf_roundtrip):** a real FIX `NewOrderSingle` -> `ExecutionReport`
round trip driven through the SAME `FixFrameParser` the engine uses, on the zf transport.
Same path/cores as the wire-floor bench (bench3 client core 40 <-> bench1 server core 50,
Solarflare ens1f0, idle-verified, 0 procs after), 141-byte FIX frame, 50k warmup + 100k
measured.

| metric | min | **p50** | p90 | p99 | p999 | max |
|---|---|---|---|---|---|---|
| FIX NOS->ER RTT over zf | 5.73 | **5.85 us** | 5.90 | 5.94 | 6.54 | 7.35 |

**Readings:**
- **FIX-frame RTT over zf = 5.85us p50** -- only ~0.4us over the 5.45us raw-zf echo (the
  FixFrameParser scan + 141B vs 128B), and **below onload's 6.66us raw-echo wire floor**.
  The FIX receive path on TCPDirect round-trips faster than an empty echo does on onload.
- Determinism carries through from the transport: **p99 just 95ns over p50** (5943 vs 5848),
  max 7.35us -- the TCPDirect signature, now on the FIX path.
- Scope: this measures transport + FIX *framing* (FixFrameParser), not full
  session-layer/field decode. Adding the ~2.3us of full-engine field/session work (measured
  separately in bench_components) lands a full-session FIX-over-zf RTT near ~8us -- vs the
  ~9.2us onload full-FIX on this fabric. The transport swap alone is the ~1.2us the wire
  floor promised, delivered end-to-end.

**Build/run:** `make zf` (defines LIBHFT_ZF, links libonload_zf; ZF_ROOT defaults to the
from-source ~/tcpdirect build, override ZF_LIB=<...libonload_zf_static.a> for a portable
binary). Both ends need `ZF_ATTR="interface=<sf-iface>"`; the server needs `--listen-host
<sf-ip>` (zf's zftl_listen rejects a wildcard INADDR_ANY bind with -EOPNOTSUPP). Everywhere
without -DLIBHFT_ZF the header is empty and the bench is a stub, so the normal tree is
unaffected (mirrors the LIBHFT_EFVI gate).

---

## Full-session FIX RTT: zf vs onload vs plain, same path (2026-07-21)

The FixSocketZf section above measured transport + framing. This is the FULL engine: the
real bench_roundtrip driven with `--transport zf` (BenchClientApp/BenchServerApp -- logon
handshake + field-level NewOrderSingle build + ExecutionReport decode), so all three
transports run the *identical* engine and bench, on the SAME bench3(client core 40)<->
bench1(server core 50) Solarflare path, nos_er scenario, 20k warmup + 100k measured,
closed-loop. (zf server = single-session ZfSingleServer after AcceptOne; plain/onload =
the normal CoreAcceptor.) Cores idle-verified; boxes swept 0-proc after.

| transport | mean | **p50** | p90 | p99 | p999 | max |
|---|---|---|---|---|---|---|
| plain (kernel TCP) | 12.76 | **12.72** | 12.92 | 13.52 | 15.59 | 102 |
| onload (busy-poll) | (skew) | **6.88** | 7.06 | 7.24 | 8.17 | 13010* |
| **TCPDirect (zf)** | 6.25 | **6.24** | 6.30 | 6.39 | 7.29 | **9.65** |

(us; *one onload iteration hit a 13ms outlier -- a safepoint/onload blip -- skewing its
mean; its p50/p99 are clean. zf's max is 9.65us, no such blip.)

**Readings:**
- **Full-engine FIX-over-zf p50 = 6.24us**: **-9% vs onload (6.88), -51% vs plain (12.72)** --
  the transport swap alone, whole engine unchanged. It sits BELOW onload's 6.66us raw-echo
  wire floor: the complete FIX round trip on TCPDirect is faster than an empty echo on onload.
- The session/field-decode layer costs only ~0.4us over the frame-only zf bench (5.85us) and
  ~0.8us over the raw zf echo (5.45us) -- much less than the ~2.3us first extrapolated.
- **Tail:** zf max 9.65us vs onload's 13ms blip and plain's 102us -- TCPDirect determinism
  holds through the full engine (p99 only 0.15us over p50).

Build/run: `make zf_session` -> bench_roundtrip_zf; `--transport zf --role server
--listen-host <sf-ip>` / `--role client --host <sf-ip>`. bench_roundtrip is now transport-
parameterised (run_client_impl<Sock>, RoundtripTaskT<Sock>); the plain FixSocket path is
byte-for-byte behaviour-preserving (guard-off build unchanged).

---

## Zero-copy RX benefit estimate -- prototype A/B before integration (2026-07-21)

Before committing to a FixSocketZf zero-copy RX integration (parser-window aliasing +
zft_zc_recv, UAF-prone), measured the ceiling with a throwaway prototype: tools/
zf_hash_pingpong.c -- a closed-loop zf ping-pong where BOTH ends FNV-1a-hash every received
byte and validate it (forces the byte-touch a plain echo skips, so the copy-vs-zero-copy
memory traffic shows in the RTT). One binary, `-C` switches copy (zft_recv) vs zero-copy
(zft_zc_recv); only RX differs. bench3<->bench1, cores 40/50, interleaved.

| payload | zerocopy p50 | copy p50 | delta/RTT | delta/receive |
|---|---|---|---|---|
| 128 B (~nos_er)   |  5736 |  5768 | 32 ns | ~16 ns |
| 512 B (~md snap)  |  7277 |  7348 | 71 ns | ~35 ns |
| 1460 B (~MSS)     | 10482 | 10553 | 71 ns | ~35 ns |

Reproducible to the ns across interleaved rounds; hash_mismatch=0 (zc reads correct bytes).
**The win PLATEAUS at ~35ns/receive (512B == 1460B)** => it is a fixed per-recv overhead
difference (zft_recv internal copy + call cost), NOT memory bandwidth. On the FIX round trip
(2 receives): ~32ns on nos_er's 6.25us (~0.5%), ~70ns on md (~0.7-1%).

**VERDICT: not worth the integration on latency grounds** (~0.5% for a UAF-prone parser
refactor). Revisit ONLY for (1) timestamp accuracy -- zft_pkt_get_timestamp gives a true RX
hw stamp vs today's software GetUtcTimeNs (a correctness win, unmeasured here); or (2)
throughput under load (this is closed-loop, 1 msg in flight; the bandwidth saving is a
throughput story). Prototype cost ~1h; saved a ~day-plus marginal integration.

---

## RX hardware-timestamp accuracy -- prototype (2026-07-21)

The zero-copy latency win is ~0.5% (above), so the real question for a FixSocketZf zero-copy
integration is the RX HARDWARE TIMESTAMP it unlocks (zft_pkt_get_timestamp is only available
on the zc_recv path -- coupling the two). tools/zf_ts_pingpong.c measures, per received echo
on the ping (bench3): gap = t_sw(clock_gettime at app-read, == today's software nicTime) -
t_hw(NIC arrival). rx_timestamping on; bench3<->bench1, 128B, 100k, 3 runs.

| run | gap mean | gap p50 | gap p99 | gap stddev | sync |
|---|---|---|---|---|---|
| 1 | 929 ns | 926 | 1000 | 42.5 ns | CLOCK_SET+IN_SYNC, 0 missing |
| 2 | 947 ns | 945 | 1016 | 37.6 ns | " |
| 3 | 939 ns | 937 | 1001 | 26.8 ns | " |

**Two findings:**
1. **Bias ~940 ns**: software RX stamps land ~940ns AFTER true NIC arrival (the RX-stack
   latency, NIC->app). Today's tick_to_trade UNDER-reports the RX portion by ~940ns (~15% of
   the 6.25us RTT).
2. **Jitter ~35 ns stddev** (p99-p50 ~70ns): the per-packet measurement noise hardware
   stamping removes. NIC clock is PTP IN_SYNC, 100% of packets stamped.

**The calibration catch:** the ~940ns bias is STABLE to ~35ns, so a one-time constant offset
(measured by this tool) added to the software stamp recovers the absolute accuracy WITHOUT
hardware stamping. So hw timestamping's IRREDUCIBLE benefit over "software + calibration
constant" is only the ~35ns per-packet jitter.

**VERDICT:** hw RX timestamping (and thus the zero-copy integration) is worth it ONLY for
EXTERNAL-reference / absolute accuracy -- true one-way latency vs a PTP-stamped feed,
correlating with exchange/venue timestamps, or regulatory wire-timestamps -- which a software
stamp CANNOT be calibrated to (IN_SYNC=1 confirms the PTP infra exists to support it). For
INTERNAL relative tick_to_trade, add the ~940ns constant and skip the integration; the ~35ns
residual doesn't justify the UAF-prone parser-window refactor.

---

## NIC timestamp vs clock_gettime -- is NICTIME on zf a hot-path SAVE? (2026-07-21)

The engine already supports `recv_ts = nicTime` instead of `clock_gettime` (the `#ifdef NICTIME`
branch in CoreInitiator/CoreAcceptor). On zf, nicTime would come from zft_pkt_get_timestamp
(zc_recv path only). So: is reading the NIC timestamp CHEAPER than the clock_gettime it
replaces? tools/zf_tscost.c times both per received packet (rdtscp-bracketed, empty bracket
subtracted), 3 runs, identical:

| call | median | p99 | net of 10ns bracket |
|---|---|---|---|
| clock_gettime(CLOCK_REALTIME) | 25.7 ns | 35-37 ns | **~15.7 ns** |
| zft_pkt_get_timestamp         | 19.3 ns | 21.4 ns | **~9.3 ns**  |

**YES -- the NIC timestamp is ~6 ns CHEAPER than clock_gettime, and TIGHTER** (p99 21 vs 37ns;
the vDSO clock hits an occasional slow path, the NIC read doesn't). So NICTIME mode on zf is a
hot-path SAVE, not a cost -- a win-win (cheaper AND the ~940ns-more-accurate stamp).

**Reframes the zero-copy total.** The zc path bundles THREE RX wins, all additive:
memcpy elimination ~16-35ns/recv (zf_hash_pingpong) + clock-call elimination ~6ns/recv (this)
+ timestamp accuracy. Combined ~22-41 ns/recv ~= 44-82ns/RTT ~= 0.7-1.3% of the 6.25us RTT --
and it's cheaper-AND-more-accurate, not a trade-off. Still modest, and the ~6ns/accuracy is
zc-path-only (zft_pkt_get_timestamp needs the zc msg), so it stays bundled with the UAF-prone
integration -- but the case is stronger than the pure-memcpy ~0.5% framing suggested.

---

## Zero-copy RX integrated: -45ns/RTT measured, matching the spike predictions (2026-07-21)

The three spikes above (memcpy ~16-35ns/recv, hw-stamp accuracy, NIC-ts ~6ns cheaper than
clock_gettime) green-lit the integration. Shipped as an OPT-IN path; the copy path stays the
default for every transport:

- **FixSocketZf::PeekZc/ConsumeZc** -- borrowed-NIC-buffer receive (zft_zc_recv/_done), with
  the packet's PTP-synced hardware RX timestamp read via zft_pkt_get_timestamp (~9.3ns, vs
  ~15.7ns clock_gettime). rx_timestamping enabled at stack alloc (graceful retry without).
- **FixFrameParser::SetWindow/DetachWindow/BufferEmpty** -- the ProcessBuffered parse+dispatch
  loop is window-location-agnostic, so the window aliases the borrow for one pass;
  DetachWindow preserves a straddling tail via the m_messageIn.m_start mid-parse rewind
  (MoveFrom's recovery pattern). Pure parser code -- unit-tested everywhere
  (test_fix_zerocopy_window, in make correctness).
- **CoreSession::ProcessZeroCopy** -- the fused step: peek -> parse+dispatch (borrow held
  through every callback) -> detach tail -> release. Falls back to the copy path while a
  partial is pending. Member template = lazily instantiated; FixSocket/Ssl untouched.
- Bench: `--zerocopy` (zf-only) drives both ends through the fused step.

**Two real bugs the harness caught before hardware:**
1. A resumable PARTIAL can consume EVERY window byte into the in-flight frame (packet ends
   right after "8=FIX.4.4<SOH>"): pointer-equality "drained" checks discard the frame.
   Both DetachWindow and BufferEmpty must consult m_messageIn.m_start. (Unit test locks it.)
2. First A/B showed only ~13ns/RTT: the zc step paid zft_pkt_get_timestamp AND still called
   GetUtcTimeNs for CheckSession -- ADDING the NIC stamp instead of replacing the clock.
   Fix: message passes reuse the stamp already in m_session.m_now. (~13 -> ~45ns, exactly
   the microbench decomposition.)

**A/B (bench3 c40 <-> bench1 c50, nos_er full session, 100k, interleaved 3x):**

| arm | p50 (3 runs) | p99 | p999 |
|---|---|---|---|
| copy (zft_recv) | 6.285 / 6.286 / 6.288 us | ~6.42 | ~7.31 |
| **zero-copy**   | **6.242 / 6.239 / 6.242** | ~6.37 | ~7.17 |

**-45ns/RTT (-0.7%), dead-stable, = the predicted memcpy(~16ns/recv) + stamp-swap(~6ns/recv)
x 2 receives.** Correctness: every run 100k full FIX sessions status=pass through the borrow
path (checksum/seqnum validation live); recv stamps are now true NIC arrival times (~940ns
earlier than the software stamp, ~35ns jitter, PTP IN_SYNC).