host hp · Historical hp (pre-2026-09-18): Xeon Gold 6154 @ 3.00 GHz · 18 cores · isolated 8–17 · 22 GB

Latency matrix: flavour × transport × acceptor model

Every cell is a FIX round trip measured on the wire between two network namespaces. Every percentile and max is the worst reported value across all sessions.

sessions 32 (multi) · 1 (single) build -march=native → skylake-avx512; gcc 14.2 (every cell re-measured 2026-09-08 under the current harness) units µs recipe v3 · open · 50 000/s aggregate (target-throughput table, swept 2026-09-09) · closed loop under Historical diagnostics · v4 captures timed since 2026-09-12 (standards 4): 30 s measured / 5 s warmup / 3 runs, each session offered its exact share of the cell aggregate, so a run measures 1 500 000 messages whatever the session count. Every measured v4 cell on this host was re-captured that way on 2026-09-12/13 and carries its window; closed and v3 keep 20 000 warmup / 20 000 measured / 3 runs · kernel cells busy-poll: SO_BUSY_POLL 50 µs on the single cells since 2026-09-09, and net.core.busy_read / busy_poll = 50 host-wide since 2026-09-10 (every kernel cell; earlier 32-session kernel rows are interrupt-moderated)

Hardware epoch boundary (2026-09-18). This is the archived Xeon Gold 6154 matrix. hp now runs an Intel Xeon Gold 6254 @ 3.10 GHz. Its topology and pin map remain compatible, but no latency, throughput or floor value below is comparable with a new run or may be extended into a current hp result. Preserve these rows as their own provenance; establish a separately labelled 6254 baseline before publishing any new hp figure. See the hardware-transition record and refresh checklist.

Both NICs are measured on this box. hp has a Mellanox pair (mlxA/mlxB, 10.9.1.x) and a Solarflare pair (sfcA/sfcB, 10.9.0.x); the Solarflare (2d:00.x) has been cabled since 2026-09-08 (a DAC between its two ports). kernel runs on the Mellanox pair (kernel TCP is NIC-agnostic), vma and socketxtreme on the Mellanox, onload and tcpdirect on the Solarflare. Cells measured before that date on onload and tcpdirect do not exist; they were unmeasurable here, not unsupported.

The kernel single cells carry a receive-path window. One TCP connection at 50 000/s over the Mellanox kernel path backs up in the client's send queue behind the NIC's interrupt moderation (about one interrupt per 80 µs at that rate): with both hot loops at 100% and the server's receive queue empty, cpp/kernel/single read a p50 of 235–847 µs. Since 2026-09-09 the single-session kernel cells set SO_BUSY_POLL 50 µs on every engine socket (KERNEL_BUSY_POLL_USECS in this box's machine file, bench-run's --busy-poll-usecs), so each non-blocking read polls the NIC queue itself; the same cell reads 15.8 µs, and 15.8 µs closed against 17.7 µs before. The 32-session kernel cells stay interrupt-driven: one run each with the window was inconclusive (mono 87 vs 99 µs, pool 101 vs 55 µs, shard 78 vs 74 µs) and no paired A/B has been run. java-jni's JNI acceptor reads its session directly under the window (its epoll_wait(0) gate ignored the option: 707 µs before, 18.6 µs after). java-pure owns its socket in Java, which has no busy-poll option, so its single cell stays on the interrupt-moderated path; the .NET Pure connection sets the option itself. The Solarflare kernel path sustains the single cell at 19–20 µs without any of this. java-pure keeps the moderated path: this JDK (17.0.20) exposes no busy-poll socket option (SO_INCOMING_NAPI_ID is read-only, TCP_QUICKACK the session already sets), and net.core.busy_read exists only in the initial network namespace, so it cannot be set for a bench namespace either. Its open kernel cells are the box's interrupt-moderated path, not a client that waits for replies: the open loop offers every due slot without consulting outstanding replies (up to 123 in flight, measured), which is what the saturated verdict is reporting.

The C++ VMA pool cell is unstable and no ring setting fixes it. The C++ pool server (cpp) over VMA stalls once per run for hundreds of milliseconds across every session. A first pairing suggested VMA's ring-per-thread allocation cured it; a second pairing inverted the result, so nothing is imposed and VMA_RING_LOGIC remains an experiment knob. The cell is published with whatever verdict it earns.

Closed and open measure different things, so they order differently. A closed session sends its next order the moment the reply arrives. An open session sends on a schedule, one order per 20 µs at 50 000/s. The v4 page shows its response time by default, measured from the scheduled slot, and publishes a cell as complete on the response CVs (since 2026-09-11). The service time, measured from the actual send, is 0.1–1.3 µs less and is the one comparable with closed, so the open figures in this paragraph are service p50s. For single cells, closed is 1–7% faster: rust-pure kernel 14.90 against 16.02 µs (open v4), onload 4.95 against 5.00, vma 5.23 against 5.40, tcpdirect 4.28 against 4.36. In the closed loop the path never idles; in the open loop each order follows up to 15 µs of idle spinning, and the client also paces and records every send. Neither cause has been isolated; a closed run with a 20 µs think time would separate the two. For 32-session cells the order reverses, by a lot: 32 closed sessions keep 32 orders in flight, so each reply queues behind the others (rust-pure onload mono: 46.1 µs closed, about 32 × the server's 1.4 µs per-order cost), while at 50 000/s aggregate an order rarely finds another queued (6.71 µs service, open v4). Closed multi-session cells therefore measure saturation, not latency.

Pools on this box (2026-09-11). Since the 2026-09-11 reboot hp isolates cores 8–17 and reserves core 8 for the pool coordinator (PIN_POOL_COORDINATOR=8), as scan reserves 19. Before that the coordinator shared worker 0's core, and the live audit, for which one /proc jiffy (10 ms of CPU in a sample) makes a second burner, refused C++-engine pools whenever their front door woke there; cpp's tcpdirect and socketxtreme pools refused to start at all. The fixes that go with the reserved core: the C++ pool front door backs off and waits for room (d98e5bc1); the .NET Native bench's status thread waits 1 s per idle pass (175932c4); the .NET Pure server starts its signal and socket runtime threads before it pins itself (22c69779) and names its pool workers (f3559a0d); the live watch retries an inconclusive sample after 1 s and scores an unreadable, exiting thread as inconclusive rather than a violation (d98e5bc1, 08fa2542). Every pool cell now passes the audit in open v4 and v3 and runs closed. Onload pool tails observed in cpp, dotnet-native and rust-pure remain in their recorded rows. The 2026-09-18 rust-pure investigation directly observed a nonblocking send sleeping on the shared Onload stack lock; counter timing identifies transient nonblocking packet-pool exhaustion as the likely entry path. This is not a fixed 600 ms spin or a first-measured-run-only event: later warmup and measured phases also stalled. The Rust row's maximum averages per-run maxima, so a ~690 ms event can appear as ~230 ms. The earlier claim that spin tuning was excluded is withdrawn: the launcher's effective settings had not been verified. A server-only EF_BUZZ_USEC=1000 experiment avoided the long wait in three diagnostic sittings, but increased p99 from about 95 to 118 µs; restoring 100 µs brought the stalls back. No default or published measurement was changed. See the evidence, limitations and future-run checklist; these hp/rust-pure closed-loop results are not a general production or cross-flavour guarantee. The reserved coordinator core did not remove this separate failure; placement remains important. dotnet-native's open v3 single cells finish in about 12 s, before the audit completes a sample in which .NET has stopped starting threads, so they are refused; their published rows come from an earlier sitting. VMA shard is noshard for every flavour: VMA 9.8.80 hands every connection on a port to the first listener registered for it.

hp has ten isolated cores (8–17) since 2026-09-11, as scan has (10–19), but its map is not scan's renumbered: core 8 is the pool coordinator's, servers take 9–12, core 13 is reserved for libvma's per-process maintenance thread and kept out of every measured server and client set, and clients take 14–17. Housekeeping is 0–7.

Historical v3 archive—not the current results. Open the current shared-epoch v4 matrix.

Applies to Target throughput and the historical diagnostic tables. Every measured cell shows p50, p99, p99.9 and max for the selected latency. Service is actual client start to completed reply; response starts at the scheduled slot and includes scheduler delay. Missing values show —. All four acceptor models stay visible; highlighting compares the selected metric among visible, qualified results.

Temporary values — do not compare flavours on them. Cells marked smoke are behaviour probes, not steady-state latency measurements. The default smoke recipe is 200 warmup / 200 measured / one run, with one session for single and four for mono, pool and shard; the table headings describe the full 32-session recipe.

Use full-recipe measurements for latency comparisons: 20 000 warmup before each run, 20 000 measured, three runs, with the stability and parity gates satisfied.

Maximum throughput

not measured Highest sustained completed request/reply rate across the whole cell, established by an open-load rate sweep. Report the passing/failing rate bracket and response latency at the passing rate. This measures the configured client/server system, not an isolated engine limit.

No validated maximum-throughput sweep is available yet. The historical closed-loop test allows only one outstanding request per session and is not maximum throughput. Existing target-rate observations are not maximum-throughput results either. Candidate-run instructions and remaining validation work.

Historical diagnostics · reply-gated RTT and pacing

These are separate experiments, retained for investigation rather than the two main throughput views. Their rates are not a backend-capacity measurement.

Historical RTT baseline · unpaced, one outstanding request per sessionService latency (µs)
Flavour Transport 1 · Single1 acceptor · 1 session 2 · Mono1 acceptor thread · 32 sessions 3 · PoolThread pool · 32 sessions 4 · ShardSharded acceptors · 32 sessions
Acceptor models measured against a 4-thread client pool — java-pure / kernel, 32 sessions Server threads on cores 10–13, client pool on 15–18, disjoint. Offered load 2000/session (64k/s aggregate), the highest rate the client sustains without backlog. Not comparable to the primary fixed-open table above — a different experiment, not a different day.
Acceptor modelp50p99p99.9maxsessions
thread-per-session (32 thr, 1 core) 2 020 9033 879 632 3 971 2635 287 85632/32
1 multiplexer 219.401706.36 2179.624752.5832/32
4-thread pool (least-loaded) 100.971534.98 7064.2610 050.5232/32

What one number means

A cell is the round-trip time of a FIX NewOrderSingle → ExecutionReport, measured on the wire between two network namespaces on one host. The clock starts at the client's scheduled send, not its actual send, so a client that falls behind is charged for the lateness instead of hiding it.

The parameters, and where they live

All of it is in one versioned file, load-bench-standards.conf, read by the wrapper, by every harness and by the checker. The common recipe is deliberately not per-scenario: the columns are comparable only if the work per message is identical in each.

commonvalue
warmup20 000, discarded
iterations20 000 measured
offered throughput50 000/s aggregate · fixed-rate open
jvm heap192 MB
scenarionos_er
per scenariosessionsacceptorsthreadsruns
1 single1113
2 mono32113
3 pool32143
4 shard3241 each3

The publication harness passes the complete recipe explicitly; bare client or launcher invocations are diagnostics, not substitutes for the harness. Every result carries bench_params (the argument vector), bench_profile (what the server configured) and bench_standards (the defaults in force, with a checksum of the file that supplied them). check-bench-params.sh refuses a table whose cells disagree, or whose profile contradicts the declared expectation.

Pinning policy is machine-independent — cores isolated, server and client sets disjoint, server from the front of the isolated set and clients from the back. The core numbers live in a per-host file, because a standard carrying one box's core list cannot be run on another, and a matrix that cannot be reproduced elsewhere is a single data point.

The four scenarios

1 single
one session, one acceptor. The engine floor, with no concurrency in it. 3 runs.
2 mono
32 sessions, one acceptor on one core. Readiness from poll(2) (or zf_muxer on TCPDirect), drained from a single poll snapshot before re-polling.
3 pool
32 sessions, one acceptor, M threads. Each session is bound to one thread for life, assigned least-loaded at accept.
4 shard
32 sessions, M independent acceptors sharing one port through SO_REUSEPORT, each on its own core, share-nothing. The kernel decides where a connection lands — there is no assignment policy. The standard uses four acceptors; M=1 is a separate diagnostic control.

3 and 4 are not the same thing. The pool and shard columns are separate across all flavours; unsupported native-stack combinations remain explicitly marked no pool / no shard.

When a cell counts

  • All 32 sessions must complete. Where fewer did, the count is printed beside the cell rather than quietly taking the median of whoever connected.
  • Every percentile and max is the worst reported value across all sessions, so one bad session cannot be averaged away.
  • A value shown as ≥ is a saturated clamp, not a reading — latency_u32() tops out at 232−1 ns.
  • Absolute numbers drift between sittings. Retained baselines and newer sweep cells may use separate artifact revisions; consult the handbook's sitting tables and source/artifact manifests before comparing them.

Core pinning

Both sides busy-poll, so a core carrying a server thread and a client starves both. Placement is therefore stated per scenario, not derived at run time.

scenarioserver coresclient cores
1 · single914
2 · mono910-12,14-17
3 · pool9-1214-17
4 · shard9-1214-17

isolated: 8-17 · VMA maintenance (reserved, no app threads): 13 · housekeeping (never measured on): 0-7 · Pool coordinator (separate from workers): 8

Server cores come from the front of the isolated set and client cores from the back, so widening the server — a pool, or more shards — eats into the middle and the two sets stay disjoint by construction rather than by arithmetic that has to be got right each time.

Unpinned is not neutral. isolcpus removes the isolated cores from every process's default affinity mask, so a thread that is never pinned does not land on a random isolated core — it lands on the housekeeping cores beside every interrupt, while the cores reserved for measurement sit idle. The JNI and .NET pool workers shipped that way for an hour; it would have read as "the pool is slow".

Not every shared core is contention. java-pure pins its accept loop and its multiplexer to the same core, and the accept loop blocks in accept() — it burns nothing. pinning-audit.sh --live samples each thread's CPU twice and reports a collision only when more than one of them actually ran.

Not every flavour can run every scenario

flavour1 single2 mono3 pool4 shard
cppyesyesyes*yes
java-pureyesyesyes*yes*
java-jniyesyesyes*yes
dotnet-nativeyesyesyes*yes
dotnet-pureyesyes*†yes*yes
rust-pureyesyesyesyes

*newly capable, not yet measured. †dotnet-pure's mono-thread cells served a thread per session — it had no multiplexed mode until now — so they are not comparable with the rest of that column. Thread-per-session with cores to spare pays no multiplexing cost, so it reads as fast rather than as different: its kernel cell sat at 26.14µs against a single-session 25.61 for exactly that reason. java-pure's mono cells are multiplexed (p99 207, not the 81 002 of its thread-per-session default); it simply had to be asked, and now the standards ask.

Three flavours now have a true pool. C++ gained FixAcceptorThreadPool — one listener, M workers, least-loaded at accept, verified serving 8/8 sessions at 2 per worker — and dotnet-pure gained the equivalent in its bench server.

java-jni reuses the very same serve pass the single-acceptor path runs, with accept disabled, so there is one serve implementation rather than two that drift; verified serving 8/8 sessions at p50 25.79µs.

rust-pure is the FIX engine written in Rust: its own client and server, no hftnet. rust-native, the Rust client driving the C++ engine through the hftnet C ABI, was parked on 2026-09-11: its server was cpp's, so its multi-session cells re-measured the C++ server. All six now pool on kernel TCP, Onload and VMA interposition. dotnet-native uses the native FixAcceptorThreadPool, with per-worker connection ids, status, errors and event rings projected through the existing managed session. Stack listener pools (TCPDirect and SocketXtreme) remain unavailable.

TCPDirect has no reuse-port listener API. SocketXtreme accepts independent SO_REUSEPORT binds, but four process pollers disconnect every client during Logon because they cannot safely own the shared completion stream. Both native-stack shard groups are marked no shard, not measured as one acceptor under a shard label.

Where the cells are not equivalent

Latency comparisons only mean something if the cells do the same work. They do not, and it is invisible in the numbers. All four end up persisting nothing, by four different mechanisms at four different costs:

flavouracceptorstoreaudit
cppFixAcceptorabsent (no template parameter)absent
cpp (--acceptor-impl runner)FixSessionRunnerAcceptornull, compile-timepresent, off
java-jniFixSessionRunnerAcceptorselected at runtime from configpresent, off
dotnet-nativeFixSessionRunnerAcceptorbranches on a null pointer per callpresent, off
dotnet-pureC# FixConnectionseparate implementation—

All flavours now declare this at runtime. Each server prints a bench_profile line stating the acceptor shape, store, audit, schedule, TLS and thread count it actually configured, so the table above is verified rather than read off the source, and check-bench-params.sh asserts it against the declared expectation.

It earns its keep. Wiring java-pure's multiplexer, its thread count was defaulted from the pool scenario's, quietly turning the mono-thread cells into a 4-thread pool. Nothing in the latency showed it; the declaration read threads=4 under a heading that says mono-thread, and that is how it was caught.

best qualified p50 for this model and selected latency > 100 µs > 1 ms — tail failure measuring run in flight not wired capability exists, no bench arm no pool / no shard transport cannot implement that shape n/a not applicable