# FIX benchmark coverage — status by language and engine

Where every engine in the [target roster](TARGET_ENGINE_ROSTER.md) has got to, for both the
microbench (Stage 2 codec) and the full bench (four flows, Stages 3–5), grouped by language so each
libhft flavour sits beside the engines it is compared with. Java and .NET each have two libhft
flavours, so those sections carry two libhft rows.

The plans for moving these cells are [PLAN_MICROBENCH.md](PLAN_MICROBENCH.md) and
[PLAN_FULL_BENCH.md](PLAN_FULL_BENCH.md).

> **Read the provenance section before quoting a number.** The four-flow cells are transcribed from
> [`COVERAGE.csv`](COVERAGE.csv), a campaign record whose evidence tree is not present on every
> developer machine — see [Provenance and the regeneration hazard](#provenance-and-the-regeneration-hazard).
> Every cell is therefore **SOURCE** grade, not **MEASURED** on the machine you are reading this on.

## Current canonical matched comparisons — 2026-10-26

These separately qualified current reports bind independently replayed full
physical originals on HP and SCAN, with the same canonical fixture and profile.

| Language | Current minimum comparison | Evidence and remaining work |
|---|---|---|
| C++ | **Pass: callback CORE / NexusFIX, all four flows on both rigs** | [Complete two-rig review](../../internal/docs/bench/RECORD_2026-10-14_cpp_callback_two_rig_comparison.md); QuickFIX-family anchor and other flavour/comparator gates remain |
| Rust | **Pass: corrected Rust / NanoFIX, first-CPU profile, all four flows on both rigs** | [Fresh physical and independent two-rig comparison](../../internal/docs/bench/RECORD_2026-10-17_rust_first_cpu_physical.md); [retired shortened clocks](../../internal/docs/bench/RECORD_2026-10-17_rust_clock_boundaries.md) remain historical; inherited-placement SCAN and TrueFix failures are retained |
| Java | Blocked: current ASCII-numeric callback-reply comparison | The [completed independent two-rig decision](../../private/docs/internal/bench/comparisons/java-ascii-numeric-callback-two-rig-20261017.json) admits HP MD/tick only and no SCAN flow; baseline/peer timing failures remain. Earlier source and eager-JVM experiments remain separate. |
| .NET | Blocked: matched peer timing; managed .NET 10 first-CPU baseline passes both rigs | The [completed independent .NET 10 decision](../../private/docs/internal/bench/comparisons/dotnet-callback-parsing-first-cpu-20261017.json) admits HP NOS/tick only. HP peer MD/mixed and every SCAN peer flow fail timing; these are completed reports. Native and earlier inherited/managed configurations retain their own grades. |

## Status by language

The generated table transcribes retained `COVERAGE.csv` campaign grades. On
2026-10-26, only the Rust open and NanoFIX rows are reconciled to **S5 pass
4/4**, using the admitted corrected complete-RTT/callback first-CPU comparison
above. The earlier S4 snapshot and shortened-clock results remain historical;
no other CSV row changes in this reconciliation. Current configuration and
matched admission remain explicitly source-bound.

Rendered by `tools/gen-bench-status` from [`engines.tsv`](engines.tsv) (who, and microbench
status) and `COVERAGE.csv` (four-flow cells). An engine with no four-flow evidence shows `not run`
rather than vanishing. Only **S5 pass** is report-grade; `diag`, `preflight` and `smoke` rows are
never a speed claim, and `non-parity` means the engine ran that flow but not in a comparable shape.

<!-- BEGIN gen-bench-status:status -->
<!-- Rendered by tools/gen-bench-status from engines.tsv and COVERAGE.csv. Do not edit by hand: change the data and rerun it. -->

### Summary: what can be compared today, by language

| Language | libhft flavours | Microbench rows (libhft / others) | Others at S5 pass | Publishable full-bench comparison |
|---|---|---|---|---|
| C++ | C++: S5 pass 4/4 | 1 / 6 | none | no: no other engine passes S5 |
| C | — | 0 / 1 | none | — (libhft has no flavour in this language) |
| Java | Java Pure: S5 pass 4/4, Java/JNI: S5 pass 4/4 | 2 / 4 | `crossfix`, `philadelphia`, `philadelphia-fast` | **yes** |
| .NET | .NET Pure: S5 pass 4/4, .NET Native: S5 pass 4/4 | 2 / 2 | none | no: no other engine passes S5 |
| Rust | Rust: S5 pass 4/4 | 1 / 5 | `nanofix` | **yes** |
| Go | — | 0 / 1 | none | — (libhft has no flavour in this language) |

### C++

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `tcp-ping-pong-cpp` | — | floor | — | — | — | — | network floor: the RTT before any FIX processing |
| **libhft-cpp** (C++) | yes | **S5 pass 4/4** | pass | pass | pass | pass | Current callback CORE/kernel/no-store comparison with NexusFIX passes all four flows on HP and SCAN after independent full replay (2026-10-14). Earlier before-parser tick comparisons remain separate; QuickFIX-family anchor, other flavours and public release remain open. |
| `fix8` | yes | S5 diag 4/4 | diag | diag | diag | diag | microbench: D/8/depth-10 W full-message codec; generated benchmark dictionary; 3/3 mutation probes passed; full bench: Original fixed-scale production Session, complete clocks/cadence/phases and immutable runtime qualify on both hosts. Both quiet DAC campaigns and independent full replay complete: HP passes all four flows; SCAN passes NOS/tick, MD misses rate and mixed misses rate/delay. Six distinct compact cells are admitted; no complete two-rig matched comparison is claimed (2026-10-24). |
| `llfix` | yes | 0/4, adapter-ready | blocked | blocked | blocked | blocked | microbench: D/8/depth-10 W receive and build through upstream LLFIX_BENCHMARK; SDK timestamp/sequence/callback/reset measured; 12 converted-field probes passed; full bench: adapter ready; retained session, store/recovery and host evidence missing |
| `nexusfix` | yes | S5 diag 4/4 | diag | diag | diag | diag | Current original NexusFIX versus callback CORE comparison passes all four flows on both accepted DAC rigs with independent replay (2026-10-14). This closes the C++ minimum only; wider-roster and release gates remain open. |
| `openfix` | excluded | 0/4, tooling-blocked | blocked | blocked | blocked | blocked | no licence; Bazel-only; gnu++23 |
| `quickfix_cpp` | yes | S5 diag 4/4 | diag | diag, non-parity | diag, non-parity | diag, non-parity | Complete canonical, bounded-preflight, threaded, checked-precision, integer-quantity, group-metadata and first-CPU physical configurations are retained separately. Both latest first-CPU campaigns and independent cold readers complete; every flow still misses the unchanged 10000/s report rate and exports zero cells (2026-10-19). Earlier non-parity rows remain historical; no unchanged timing retry is pending. |
| `robaho_cpp_fix_engine` | excluded | 0/4, compile-failed | blocked | blocked | blocked | blocked | upstream build fails; licence unknown |
| `fixpp` | yes | codec only | — | — | — | — |  |
| `hffix` | yes | codec only | — | — | — | — |  |
| `robaho_cpp_fix_codec` | excluded | codec only | — | — | — | — | microbench: no licence at 4a95e7c: Stage 2 adapter removed; archived timings withheld; full bench: no tested session endpoint |

### C

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `libtrading` | yes | S5 diag 4/4 | diag | diag, non-parity | diag, non-parity | diag, non-parity | microbench: public numeric-tag parser/unparser; depth-10 W requires the existing 128-field capacity patch; full bench: Separate canonical text-numeric configuration passes all four report flows on both accepted DAC rigs with exact independent replay and observed before/after host/NIC provenance. Eight compact cells retain source CDFs; website delivery uses summaries. The explicit 48-to-128 field capacity patch and older non-parity adapter remain separate (2026-10-19). C is reference-only, with no LibHFT C flavour or matched-language admission. |

### Java

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `tcp-ping-pong-java` | — | floor | — | — | — | — | network floor: the RTT before any FIX processing |
| **libhft-java-pure** (Java Pure) | yes | **S5 pass 4/4** | pass | pass | pass | pass |  |
| **libhft-java-jni** (Java/JNI) | yes | **S5 pass 4/4** | pass | pass | pass | pass |  |
| `artio` | yes | S5 diag 4/4 | diag | diag | diag | diag | microbench: source-owned D/8/W dictionary generation; retained host smoke/parity required; full bench: New original numeric codec and production Session qualify through clocks, bounded capture, cadence/backpressure, shared phases and installed DAC runtime on both hosts. Both full sixteen-pair quiet physical campaigns and independent full-value admissions complete; all flows retain rate/delay failures and zero cells are exported (2026-10-25). Original v10 diagnostics remain historical; no unchanged timing retry is pending. |
| `crossfix` | excluded | **S5 pass 4/4** | pass | pass | pass | pass | microbench: its obfuscated buffer API cannot be reset and reparsed fairly; full bench: CrossFIX 1.6; 2.0 is not headline-grade; licensed installation only |
| `falcon` | partial | S3 smoke 1/4 | smoke | unsupported | unsupported | unsupported | microbench: D and 8 only: no V/W/X market-data types; full bench: no market-data messages, so N/A for the four flows |
| `philadelphia` | yes | **S5 pass 4/4** | pass | pass | pass | pass |  |
| `philadelphia-fast` | — | **S5 pass 4/4** | pass | pass | pass | pass | microbench: a transport configuration of philadelphia, not a separate codec; full bench: Current complete callback comparison with the ASCII-numeric Java Pure baseline admits HP MD/tick only; no SCAN flow is admitted, so the Java two-rig minimum is blocked (2026-10-17). Both full physical collections and independent readers complete. Historical passing cells and superseded source comparisons never fill a current failed flow. |
| `quickfixj` | yes | S5 diag 4/4 | diag | diag | diag | diag | Original SDK canonical callback/complete-RTT default-GC configuration completes both sixteen-pair physical matrices and independent bounded collections. Every flow still fails rate and scheduler delay; SCAN NOS also fails percentile stability. Prior Epsilon, race and qualification failures remain separate; no unchanged timing retry is pending. QuickFIX-family anchor admission remains blocked (2026-10-17). |

### .NET

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `tcp-ping-pong-net` | — | floor | — | — | — | — | network floor: the RTT before any FIX processing |
| **libhft-dotnet-managed** (.NET Pure) | yes | **S5 pass 4/4** | pass | pass | pass | pass | Actual public managed .NET 10 first-CPU complete-RTT/callback baseline passes all four flows on both accepted DAC rigs with independent full replay (2026-10-17). The matched QuickFIX/n comparison remains blocked by peer timing. Earlier .NET 8 and inherited-placement configurations keep their own source/clock identities and grades. |
| **libhft-dotnet-native** (.NET Native) | yes | **S5 pass 4/4** | pass | pass | pass | pass | Actual public Native handler/complete-RTT first-CPU configuration passes all four flows on both accepted DAC rigs after the qualified single-traversal book extraction and independent full replay (2026-10-17). Inherited-placement, historical receive-clock and earlier mixed/tick failures remain separate. No .NET peer comparison is inferred from Native results. |
| `fixantenna_net` | yes | S5 diag 4/4 | diag | diag | diag | diag | microbench: public RawFixUtil/Message codec; source pinned ff38f51; stage-2 parity passed; full bench: Original numeric writer, production Session, exact clocks/shared phases, report preflight and immutable DAC deployments qualify independently on both hosts. Both original report and separate receiver-placement full physical campaigns complete with independent replay; all flows fail rate/delay and write zero cells (2026-10-26). Tick retains an SDK-envelope prepared-application boundary blocker; original SDK/harness stay unchanged. No unchanged campaign rerun or .NET admission is pending. |
| `quickfix_n` | yes | S5 diag 4/4 | diag | diag | diag | diag | Original public SDK .NET 10 parsing-metadata/complete-request first-CPU configuration completes both quiet physical campaigns and independent full replay. HP NOS/tick pass; HP MD/mixed and every SCAN flow fail rate, with other delay/stability blockers retained. The independent two-rig .NET comparison is blocked and failed flows have no ratios (2026-10-17). Validated/inherited and historical non-parity configurations remain separate. |

### Rust

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `tcp-ping-pong-rust` | — | floor | — | — | — | — | closed-loop TCP RTT before FIX processing; bench/rust/ping-pong |
| **libhft-rust-pure** (Rust) | yes | **S5 pass 4/4** | pass | pass | pass | pass | Corrected complete-RTT/callback first-CPU Rust open versus NanoFIX comparison passes all four flows on HP and SCAN with independently replayed physical evidence (2026-10-17). Earlier shortened-clock canonical comparisons are retired; inherited-placement failures, TrueFix, other flavours and release remain separate. |
| **libhft-rust-native** (Rust over the C ABI) | parked | parked | — | — | — | — | parked 2026-09-11 |
| `dfx` | yes | S5 diag 4/4 | diag | diag, non-parity | diag, non-parity | diag, non-parity | Both original pinned SDK writer/parser investigations preserve all values and group associations but expose canonical outer-field-order failures in every flow. Explicit body-only order controls pass; the production Session uses the default writer. A supported canonical production writer remains blocked before fresh Session/clock/DAC admission (2026-10-19); legacy non-parity diagnostics remain historical. |
| `fixer_rs` | yes | S5 diag 4/4 | diag | diag, non-parity | diag, non-parity | diag, non-parity | Separate public custom-order SDK configuration qualifies through real Session, prepared/callback clocks and timed phases on both hosts. Both complete quiet DAC matrices and independent full replay finish: HP NOS/tick pass; HP MD/mixed and all SCAN flows retain report blockers. Two new compact cells are admitted; no unchanged timing retry is pending. A complete matched comparison remains blocked (2026-10-23). |
| `nanofix` | yes | **S5 pass 4/4** | pass | pass | pass | pass | Corrected complete-RTT/callback first-CPU original public-transport NanoFIX versus Rust open comparison passes all four flows on both accepted DAC rigs with independent matched replay (2026-10-17). Older shortened-clock results remain retired and do not supply current admission; the explicit Aeron compatibility patch remains declared. |
| `truefix` | yes | S4 diag 4/4 | S4 diag canon | S4 diag canon | S4 diag canon | S4 diag canon | Corrected callback/complete-RTT current-thread production Session completes both full physical matrices and independent cold readers (2026-10-17). Five current compact cells pass: HP NOS/MD/tick and SCAN NOS/tick. HP mixed and SCAN MD/mixed retain report timing failures. Original shortened-clock results remain retired; a compatible separately qualified first-CPU/configuration and fresh matched two-rig comparison remain open. |
| `ferrumfix` | yes | codec only | — | — | — | — | through the pinned v0.7.0 shim |

### Go

| Engine | Microbench | Full bench | nos_er | md | md_nos_er | tick_to_trade | Notes |
|---|---|---|---|---|---|---|---|
| `quickfix_go` | yes | S5 diag 4/4 | diag | diag | diag | diag | Original public writer/parser capability completes on both hosts at the retained microbench SDK pin, with all values and group associations intact. All 240 frames per host differ in canonical outer order because the public field map sorts tags; fresh Session/clock/DAC admission is blocked on a supported writer path (2026-10-19). Go remains reference-only, with no LibHFT Go flavour. |
<!-- END gen-bench-status:status -->

---

## How to read the status

"Bench flavor" is two independent axes: a full-bench cell is a *(stage, scenario)* pair. The
microbench is Stage 2 alone; its "yes" means the Stage-2 launcher runs a fair adapter and it passed a
smoke run. The [2026-09-29 campaign](RECORD_2026-09-29_stage2-microbench.md)
provides host-local timing and parity evidence; inspect each report's build
and parity tables before quoting an engine's numbers. The robaho codec timing
cells are withheld because its pinned source has no licence grant.

### Axis A — stages (increasing rigour)

From [`CAMPAIGN_TEST_PLAN.md`](CAMPAIGN_TEST_PLAN.md).
Each stage is a gate: an engine that fails one does not reach the next.

| Stage | Name | What it establishes |
|---|---|---|
| 0 | Candidate registry | the engine exists, is licensed usably, and is fetchable |
| 1 | Build and upstream test | it compiles and its own test suite passes |
| 2 | Parser / builder microbench | encode and decode cost, no session |
| 3 | Session capability smoke | it can log on and exchange the scenario at all |
| 4 | Realistic workflow benchmark | latency under the canonical scenarios, single machine |
| 5 | Cross-machine Linux benchmark | latency over a real NIC between two hosts |
| 6 | Cross-platform portability | it runs beyond the primary platform |
| 7 | Correctness and conformance | it is *right*, not merely fast |

Stage 5 is the one that produces quotable numbers. Stages 2 and 3 produce signal; stage 4 is
single-machine and therefore loopback-contaminated.

**Stage 5 has two grades, and conflating them is the most common misreading of this matrix:**

- **`S5 pass`** — report-grade. Full sample count, cross-machine, parity-checked.
- **`S5 diag`** — diagnostic only. Ran and produced timings, but sample count or parity is
  insufficient to quote. *Not* a pass.

`non-parity` on a scenario means the engine ran it but not in a shape comparable to the others,
so cross-engine comparison of that cell is invalid.

### Axis B — canonical scenarios

The authoritative contract is `CONTRACT_SCENARIOS.md`. Four scenarios, and only four —
earlier rows (`md_er`, `business`, `nos_md_er`, `nos_md_er_business`) are retired and folded in.

| Scenario | Wire exchange | What it stresses |
|---|---|---|
| `nos_er` | `35=D → 35=8` | order entry round trip |
| `md` | `35=V → 35=W` | market-data request and snapshot |
| `md_nos_er` | `35=V → 35=W → 35=D → 35=8` | mixed session, both flows interleaved |
| `tick_to_trade` | server-side `35=W → 35=D` | the number that actually matters commercially |

**4 scenarios × 8 stages = the full flavor grid.** The tables above report the scenario axis at
the highest stage each engine reached, because that is the only summary that does not overstate.

---

## Name reconciliation

Three name-spaces exist for the same engines. This is why coverage reports disagree.

| CSV matrix | `DECLARED_ENGINES` | Resolution |
|---|---|---|
| `libhft-cpp` | `libhft` | same engine |
| `libhft-java-jni` | `libhft-java-jni` | same engine |
| `libhft-java-pure` | `libhft-java-pure` | same engine |
| `libhft-dotnet-managed` | `libhft-net-managed` / Stage-5 `libhft-net` | managed C# row; legacy Stage-5 executable still emits `libhft-net` |
| `libhft-dotnet-native` | hftnet/provider-smoke | native C++-backed .NET row |
| `quickfixj` | `quickfix_j` | same engine, punctuation differs |

`DECLARED_ENGINES` now includes the full-bench targets, including CrossFIX,
Philadelphia-fast, FIX Antenna, TrueFix and NanoFIX. The codec-only and
network-floor rows still come from the roster rather than this full-engine
list; their absence from it is intentional.

## Provenance and the regeneration hazard

`COVERAGE.csv` is *generated*, by
[`generate_fix_benchmark_engine_coverage.py`](../../bench/cpp/fixbench/tools/generate_fix_benchmark_engine_coverage.py),
from an evidence tree at `C:/Temp/libhft/open-source-fix-engines`.

**That tree is not on every developer machine.** On a box with a partial tree, regenerating turns
most of this matrix to `0/4 / none` — the engines have not regressed, the evidence is simply
absent locally. Two consequences:

- **Do not treat a locally-regenerated matrix as a result.** Check the evidence tree size first.
  A `results/` directory with a handful of entries cannot support a 28-engine matrix.
- **Do not regenerate over `docs/`.** The generator's default `--output` is `COVERAGE.csv`,
  a path that no longer exists. Redirecting it at the live `docs/` copy on a machine with a partial
  tree overwrites a full campaign record with an empty one, and the loss is silent.

The explicit 2026-10-26 Rust/NanoFIX reconciliation reads the committed
[corrected two-rig decision](../../private/docs/internal/bench/comparisons/rust-callback-first-cpu-20261017.json)
(SHA-256 `9af1f97d8c628afbf12ce2e6776d114fb9e15fc72f0b3dd8ce0b51be2a192df8`).
All sixteen current result cells match its original per-rig p50/p99/p99.9/worst,
source and generation bindings, 30,000 samples, three passing report rates and
the website's admitted publication decision. Exactly two CSV rows are updated;
the full evidence-discovery generator is not run. This is a recorded-admission
update, with no new benchmark or promotion of a retired clock configuration.

The status of every cell above is therefore **SOURCE** — transcribed from the
committed campaign record or the explicit admitted-row reconciliation — not
**MEASURED** on the machine you are reading this on.

## Status of this document

Written 2026-08-14 as a hand-transcribed matrix; reorganised by language on 2026-09-29, when the
tables became `tools/gen-bench-status` output and the roster, open work and "what the matrix says"
moved to [TARGET_ENGINE_ROSTER.md](TARGET_ENGINE_ROSTER.md), the two plans and the rendered summary.
The stage and scenario definitions above were extracted from `CAMPAIGN_TEST_PLAN.md` and
`CONTRACT_SCENARIOS.md`; the name reconciliation is still hand-written and is the section most
likely to rot.