Every cell is a FIX round trip measured on the wire between two network namespaces. Every percentile and max is the worst reported value across all sessions.
Hardware epoch boundary (2026-09-18). This is the archived Xeon Gold 6154 matrix. hp now runs an Intel Xeon Gold 6254 @ 3.10 GHz. Its topology and pin map remain compatible, but no latency, throughput or floor value below is comparable with a new run or may be extended into a current hp result. Preserve these rows as their own provenance; establish a separately labelled 6254 baseline before publishing any new hp figure. See the hardware-transition record and refresh checklist.
Both NICs are measured on this box. hp has a Mellanox pair (mlxA/mlxB, 10.9.1.x)
and a Solarflare pair (sfcA/sfcB, 10.9.0.x); the Solarflare (2d:00.x) has been cabled
since 2026-09-08 (a DAC between its two ports). kernel runs on the Mellanox pair (kernel TCP is
NIC-agnostic), vma and socketxtreme on the Mellanox, onload and tcpdirect on the
Solarflare. Cells measured before that date on onload and tcpdirect do not exist; they were unmeasurable
here, not unsupported.
The kernel single cells carry a receive-path window. One TCP connection at 50 000/s over the Mellanox
kernel path backs up in the client's send queue behind the NIC's interrupt moderation (about one interrupt per
80 µs at that rate): with both hot loops at 100% and the server's receive queue empty, cpp/kernel/single read
a p50 of 235–847 µs. Since 2026-09-09 the single-session kernel cells set SO_BUSY_POLL 50 µs on
every engine socket (KERNEL_BUSY_POLL_USECS in this box's machine file, bench-run's
--busy-poll-usecs), so each non-blocking read polls the NIC queue itself; the same cell reads
15.8 µs, and 15.8 µs closed against 17.7 µs before. The 32-session kernel cells stay interrupt-driven: one run
each with the window was inconclusive (mono 87 vs 99 µs, pool 101 vs 55 µs, shard 78 vs 74 µs) and no paired
A/B has been run. java-jni's JNI acceptor reads its session directly under the window (its epoll_wait(0)
gate ignored the option: 707 µs before, 18.6 µs after). java-pure owns its socket in Java, which has no busy-poll
option, so its single cell stays on the interrupt-moderated path; the .NET Pure connection sets the option itself.
The Solarflare kernel path sustains the single cell at 19–20 µs without any of this. java-pure keeps the
moderated path: this JDK (17.0.20) exposes no busy-poll socket option (SO_INCOMING_NAPI_ID is
read-only, TCP_QUICKACK the session already sets), and net.core.busy_read exists only in
the initial network namespace, so it cannot be set for a bench namespace either. Its open kernel cells are the
box's interrupt-moderated path, not a client that waits for replies: the open loop offers every due slot without
consulting outstanding replies (up to 123 in flight, measured), which is what the saturated verdict is reporting.
The C++ VMA pool cell is unstable and no ring setting fixes it. The C++ pool server (cpp) over VMA stalls once per run for hundreds of milliseconds across every session. A first pairing suggested
VMA's ring-per-thread allocation cured it; a second pairing inverted the result, so nothing is imposed and
VMA_RING_LOGIC remains an experiment knob. The cell is published with whatever verdict it earns.
Closed and open measure different things, so they order differently. A closed session sends its next order the moment the reply arrives. An open session sends on a schedule, one order per 20 µs at 50 000/s. The v4 page shows its response time by default, measured from the scheduled slot, and publishes a cell as complete on the response CVs (since 2026-09-11). The service time, measured from the actual send, is 0.1–1.3 µs less and is the one comparable with closed, so the open figures in this paragraph are service p50s. For single cells, closed is 1–7% faster: rust-pure kernel 14.90 against 16.02 µs (open v4), onload 4.95 against 5.00, vma 5.23 against 5.40, tcpdirect 4.28 against 4.36. In the closed loop the path never idles; in the open loop each order follows up to 15 µs of idle spinning, and the client also paces and records every send. Neither cause has been isolated; a closed run with a 20 µs think time would separate the two. For 32-session cells the order reverses, by a lot: 32 closed sessions keep 32 orders in flight, so each reply queues behind the others (rust-pure onload mono: 46.1 µs closed, about 32 × the server's 1.4 µs per-order cost), while at 50 000/s aggregate an order rarely finds another queued (6.71 µs service, open v4). Closed multi-session cells therefore measure saturation, not latency.
Pools on this box (2026-09-11). Since the 2026-09-11 reboot hp isolates cores 8–17 and reserves core 8
for the pool coordinator (PIN_POOL_COORDINATOR=8), as scan reserves 19. Before that the coordinator shared
worker 0's core, and the live audit, for which one /proc jiffy (10 ms of CPU in a sample) makes a second burner,
refused C++-engine pools whenever their front door woke there; cpp's tcpdirect and socketxtreme pools refused to start
at all. The fixes that go with the reserved core: the C++ pool front door backs off and waits for room (d98e5bc1); the
.NET Native bench's status thread waits 1 s per idle pass (175932c4); the .NET Pure server starts its signal and
socket runtime threads before it pins itself (22c69779) and names its pool workers (f3559a0d); the live watch retries
an inconclusive sample after 1 s and scores an unreadable, exiting thread as inconclusive rather than a violation
(d98e5bc1, 08fa2542). Every pool cell now passes the audit in open v4 and v3 and runs closed.
Onload pool tails observed in cpp, dotnet-native and rust-pure remain in their recorded rows.
The 2026-09-18 rust-pure investigation directly observed a nonblocking send sleeping on the
shared Onload stack lock; counter timing identifies transient nonblocking packet-pool exhaustion
as the likely entry path. This is not a fixed 600 ms spin or a first-measured-run-only event:
later warmup and measured phases also stalled. The Rust row's maximum averages per-run maxima,
so a ~690 ms event can appear as ~230 ms. The earlier claim that spin tuning was excluded is
withdrawn: the launcher's effective settings had not been verified. A server-only
EF_BUZZ_USEC=1000 experiment avoided the long wait in three diagnostic sittings,
but increased p99 from about 95 to 118 µs; restoring 100 µs brought the stalls back.
No default or published measurement was changed. See the
evidence, limitations and future-run checklist;
these hp/rust-pure closed-loop results are not a general production or cross-flavour guarantee.
The reserved coordinator core did not remove this separate failure; placement remains important.
dotnet-native's open v3 single cells finish
in about 12 s, before the audit completes a sample in which .NET has stopped starting threads, so they are refused;
their published rows come from an earlier sitting. VMA shard is noshard for every flavour: VMA 9.8.80
hands every connection on a port to the first listener registered for it.
hp has ten isolated cores (8–17) since 2026-09-11, as scan has (10–19), but its map is not scan's renumbered: core 8 is the pool coordinator's, servers take 9–12, core 13 is reserved for libvma's per-process maintenance thread and kept out of every measured server and client set, and clients take 14–17. Housekeeping is 0–7.
Historical v3 archive—not the current results. Open the current shared-epoch v4 matrix.
Applies to Target throughput and the historical diagnostic tables. Every measured cell shows p50, p99, p99.9 and max for the selected latency. Service is actual client start to completed reply; response starts at the scheduled slot and includes scheduler delay. Missing values show —. All four acceptor models stay visible; highlighting compares the selected metric among visible, qualified results.
Temporary values — do not compare flavours on them. Cells marked smoke are behaviour probes, not steady-state latency measurements. The default smoke recipe is 200 warmup / 200 measured / one run, with one session for single and four for mono, pool and shard; the table headings describe the full 32-session recipe.
Use full-recipe measurements for latency comparisons: 20 000 warmup before each run, 20 000 measured, three runs, with the stability and parity gates satisfied.
not measured Highest sustained completed request/reply rate across the whole cell, established by an open-load rate sweep. Report the passing/failing rate bracket and response latency at the passing rate. This measures the configured client/server system, not an isolated engine limit.
No validated maximum-throughput sweep is available yet. The historical closed-loop test allows only one outstanding request per session and is not maximum throughput. Existing target-rate observations are not maximum-throughput results either. Candidate-run instructions and remaining validation work.
These are separate experiments, retained for investigation rather than the two main throughput views. Their rates are not a backend-capacity measurement.
| Flavour | Transport | 1 · Single1 acceptor · 1 session | 2 · Mono1 acceptor thread · 32 sessions | 3 · PoolThread pool · 32 sessions | 4 · ShardSharded acceptors · 32 sessions |
|---|
| Flavour | Transport | 1 · Single1 acceptor · 1 session | 2 · Mono1 acceptor thread · 32 sessions | 3 · PoolThread pool · 32 sessions | 4 · ShardSharded acceptors · 32 sessions |
|---|
BENCH-CONTRACT §1 mode paced: each session is paced at the cell aggregate divided by its session count; the rate beneath each model's metrics is achieved/target and is flagged when the cell could not offer the load it was asked for. Above capacity a paced row is a saturation result, not a latency row.
| Flavour | Transport | 1 · Single1 acceptor · 1 session | 2 · Mono1 acceptor thread · 32 sessions | 3 · PoolThread pool · 32 sessions | 4 · ShardSharded acceptors · 32 sessions |
|---|
Use Latency to switch between Service (default) and Response. Each cell shows p50 / p99 / p99.9 / max in µs; Rate is achieved / target requests/s per session. Missing historical response metrics show —, never service values or zero. Rates and qualifications are unchanged by the selection.
Response p99.9 and max were recovered for all 59 retained measured OPEN cells from their original validated client logs. Existing values and verdicts are unchanged; this is publication repair, not a new benchmark. Recovery evidence.
BENCH-CONTRACT §1 mode open: fixed send schedule, independent of replies. How to run this bench. Response runs from the intended request-start slot to completed reply, including scheduler delay. Service runs from the client's actual request start to completed reply: the whole client/network/server exchange, not server processing alone. For each request, response = scheduler delay + service. The target is shared: 50 000/s for one session, or floor(50 000/32) = 1 562/s per session (49 984/s effective aggregate) at 32 sessions. Rate annotations are per session, using the slowest session's achieved rate. Blank response metrics mean unrecorded, never zero. Cells are replaced as the current sweep completes; see all current/retained v3 sitting tables documented in the handbook for progress and provenance. Cells listed in any of those tables have v3 evidence, subject to their reported status; invalidated observations are withdrawn, not usable measurements. C++/.NET open values that included warmup in the extraction sink were replaced by corrected reruns. Values absent from every table are historical. Retained baselines and newer sweep cells can use separate artifact revisions: combined coverage is not one identical-binary sitting. Closed and paced baselines are retained under Historical diagnostics, not relabelled as either throughput experiment.
Publication reconciled: retained sitting evidence restores
59 full-delivery cells: 10 ok and 49 partial / unstable.
Available partial timings are shown with an explicit qualifier and never receive best-result
highlighting. One genuine Java/JNI / VMA / pool failure remains; all 12 .NET Pure
cells are withdrawn for missing client CPU-pin application. See the
source-by-source reconciliation.
32-session pacing limitation: all three multi-session layouts currently align each process's eight session deadlines, producing batches roughly every 640.204 µs rather than uniformly spaced aggregate sends. Catch-up can add further bursts, and the four processes do not share a common schedule origin. The target and achieved rates are averages, not spacing checks. These v3 values describe that bursty workload; they must not be presented as measurements of a uniformly staggered stream. Offline replay of all 15 kernel flavour/layout combinations shows local median gaps of roughly 5–7 µs and p90 gaps of 585–601 µs, versus 80.026 µs for an evenly spaced client process. See the replay scope and limitations.
.NET integrity correction and pinning investigation: full request ownership and reply-stage guards now prevent duplicate or malformed replies from reusing timestamps. All 165 .NET tests pass. Existing latency rows predate this change; no old values are relabelled as corrected measurements. An installed eight-cell kernel smoke delivered fully but exposed missing CPU-pin application in the managed open/multi client. Archived-source proof confirms that all twelve managed OPEN latency/rate values require withdrawal; any retained numbers for those cells are invalid pending fresh pinned runs, not evidence of delivery failures. Their original full-delivery evidence is retained. Smoke timings are not imported; the pacing policy is unchanged. See verification details.
Fresh installed .NET verification is tracked after each kernel smoke cell; these short-run timings do not replace full-size target-throughput values.
Remaining open-load failure: Java/JNI/VMA pool completed 16/32 sessions in R10. Separate logging and authorized statistics diagnostics reproduced the failure; their timings are excluded. The statistics run completed 24/32 sessions and narrowed its failed client's deficit to transmit/progress before server consumption, without proving the cause. See the handbook for the retained evidence and limitations.
Recovery comparison completed: separate ten-second-grace and matched two-second-control JNI/VMA pool sittings both delivered 32/32 sessions, with stability qualified and no watchdog timeouts. Identical installed binaries were used for the pair. There is no evidence that longer patience fixed the intermittent stall; the canonical watchdog stays at two seconds. These diagnostics do not replace matrix values.
.NET Native shard correction: the R7 kernel/Onload shard numbers were withdrawn after logs and code showed only shard 0 was started. All 32 connections were served by that one acceptor; delivery counts alone did not validate the requested four-shard topology. All three .NET Native shard transports have now been rerun after repair. In R10 VMA, all four listeners started, but admission was 32/0/0/0: that result does not demonstrate four-way traffic distribution or sharding speedup.
Pinning-audit limitation: the earlier continuous watcher discarded an audit-script error, so empty watch logs do not prove clean runtime placement. Launcher/VMA affinity records remain available. Corrected sampling starts at 02:58:01 UTC during R7 Java/JNI/Onload/pool; earlier coverage is inconclusive. Low-duty activity on the core shared by the pool accept thread and worker 0 has been flagged. R10's watcher recorded 106 PASS, 24 VIOLATION and 13 INCONCLUSIVE samples; it does not establish uniformly clean placement. See the handbook for details.
| Acceptor model | p50 | p99 | p99.9 | max | sessions |
|---|---|---|---|---|---|
| thread-per-session (32 thr, 1 core) | 2 020 903 | 3 879 632 | 3 971 263 | 5 287 856 | 32/32 |
| 1 multiplexer | 219.40 | 1706.36 | 2179.62 | 4752.58 | 32/32 |
| 4-thread pool (least-loaded) | 100.97 | 1534.98 | 7064.26 | 10 050.52 | 32/32 |
A cell is the round-trip time of a FIX NewOrderSingle → ExecutionReport, measured on the wire between two network namespaces on one host. The clock starts at the client's scheduled send, not its actual send, so a client that falls behind is charged for the lateness instead of hiding it.
All of it is in one versioned file, load-bench-standards.conf, read
by the wrapper, by every harness and by the checker. The common recipe is
deliberately not per-scenario: the columns are comparable only if the work
per message is identical in each.
| common | value |
|---|---|
| warmup | 20 000, discarded |
| iterations | 20 000 measured |
| offered throughput | 50 000/s aggregate · fixed-rate open |
| jvm heap | 192 MB |
| scenario | nos_er |
| per scenario | sessions | acceptors | threads | runs |
|---|---|---|---|---|
| 1 single | 1 | 1 | 1 | 3 |
| 2 mono | 32 | 1 | 1 | 3 |
| 3 pool | 32 | 1 | 4 | 3 |
| 4 shard | 32 | 4 | 1 each | 3 |
The publication harness passes the complete recipe explicitly;
bare client or launcher invocations are diagnostics, not substitutes for the harness.
Every result carries bench_params (the argument vector),
bench_profile (what the server configured) and
bench_standards (the defaults in force, with a checksum of the file
that supplied them). check-bench-params.sh refuses a table whose
cells disagree, or whose profile contradicts the declared expectation.
Pinning policy is machine-independent — cores isolated, server and client sets disjoint, server from the front of the isolated set and clients from the back. The core numbers live in a per-host file, because a standard carrying one box's core list cannot be run on another, and a matrix that cannot be reproduced elsewhere is a single data point.
poll(2) (or zf_muxer on TCPDirect), drained from a
single poll snapshot before re-polling.SO_REUSEPORT, each on its own core, share-nothing. The
kernel decides where a connection lands — there is no assignment policy.
The standard uses four acceptors; M=1 is a separate diagnostic control.3 and 4 are not the same thing. The pool and shard columns are separate across all flavours; unsupported native-stack combinations remain explicitly marked no pool / no shard.
≥ is a saturated clamp, not a reading —
latency_u32() tops out at 232−1 ns.Both sides busy-poll, so a core carrying a server thread and a client starves both. Placement is therefore stated per scenario, not derived at run time.
| scenario | server cores | client cores |
|---|---|---|
| 1 · single | 9 | 14 |
| 2 · mono | 9 | 10-12,14-17 |
| 3 · pool | 9-12 | 14-17 |
| 4 · shard | 9-12 | 14-17 |
isolated: 8-17 · VMA maintenance (reserved, no app threads): 13 · housekeeping (never measured on): 0-7 · Pool coordinator (separate from workers): 8
Server cores come from the front of the isolated set and client cores from the back, so widening the server — a pool, or more shards — eats into the middle and the two sets stay disjoint by construction rather than by arithmetic that has to be got right each time.
Unpinned is not neutral. isolcpus removes the
isolated cores from every process's default affinity mask, so a thread that is
never pinned does not land on a random isolated core — it lands on the
housekeeping cores beside every interrupt, while the cores reserved for
measurement sit idle. The JNI and .NET pool workers shipped that way for an hour;
it would have read as "the pool is slow".
Not every shared core is contention. java-pure pins its accept loop
and its multiplexer to the same core, and the accept loop blocks in
accept() — it burns nothing.
pinning-audit.sh --live samples each thread's CPU twice and reports a
collision only when more than one of them actually ran.
| flavour | 1 single | 2 mono | 3 pool | 4 shard |
|---|---|---|---|---|
| cpp | yes | yes | yes* | yes |
| java-pure | yes | yes | yes* | yes* |
| java-jni | yes | yes | yes* | yes |
| dotnet-native | yes | yes | yes* | yes |
| dotnet-pure | yes | yes*† | yes* | yes |
| rust-pure | yes | yes | yes | yes |
*newly capable, not yet measured. †dotnet-pure's mono-thread cells served a thread per session — it had no multiplexed mode until now — so they are not comparable with the rest of that column. Thread-per-session with cores to spare pays no multiplexing cost, so it reads as fast rather than as different: its kernel cell sat at 26.14µs against a single-session 25.61 for exactly that reason. java-pure's mono cells are multiplexed (p99 207, not the 81 002 of its thread-per-session default); it simply had to be asked, and now the standards ask.
Three flavours now have a true pool. C++ gained
FixAcceptorThreadPool — one listener, M workers, least-loaded
at accept, verified serving 8/8 sessions at 2 per worker — and
dotnet-pure gained the equivalent in its bench server.
java-jni reuses the very same serve pass the single-acceptor path runs, with accept disabled, so there is one serve implementation rather than two that drift; verified serving 8/8 sessions at p50 25.79µs.
rust-pure is the FIX engine written in Rust: its own client and server, no hftnet.
rust-native, the Rust client driving the C++ engine through the hftnet C ABI, was parked on
2026-09-11: its server was cpp's, so its multi-session cells re-measured the C++ server.
All six now pool on kernel TCP, Onload and VMA interposition. dotnet-native uses the
native FixAcceptorThreadPool, with per-worker connection ids, status,
errors and event rings projected through the existing managed session. Stack
listener pools (TCPDirect and SocketXtreme) remain unavailable.
TCPDirect has no reuse-port listener API. SocketXtreme accepts
independent SO_REUSEPORT binds, but four process pollers disconnect
every client during Logon because they cannot safely own the shared completion
stream. Both native-stack shard groups are marked no shard, not measured
as one acceptor under a shard label.
Latency comparisons only mean something if the cells do the same work. They do not, and it is invisible in the numbers. All four end up persisting nothing, by four different mechanisms at four different costs:
| flavour | acceptor | store | audit |
|---|---|---|---|
| cpp | FixAcceptor | absent (no template parameter) | absent |
| cpp (--acceptor-impl runner) | FixSessionRunnerAcceptor | null, compile-time | present, off |
| java-jni | FixSessionRunnerAcceptor | selected at runtime from config | present, off |
| dotnet-native | FixSessionRunnerAcceptor | branches on a null pointer per call | present, off |
| dotnet-pure | C# FixConnection | separate implementation | — |
All flavours now declare this at runtime. Each server
prints a bench_profile line stating the acceptor shape, store, audit,
schedule, TLS and thread count it actually configured, so the table above is
verified rather than read off the source, and
check-bench-params.sh asserts it against the declared expectation.
It earns its keep. Wiring java-pure's multiplexer, its thread count
was defaulted from the pool scenario's, quietly turning the mono-thread
cells into a 4-thread pool. Nothing in the latency showed it; the declaration read
threads=4 under a heading that says mono-thread, and that is how it
was caught.