Latency matrix: flavour × transport × acceptor model

Historical: Java-pure thread-per-session contention

An earlier harness ran 32 busy-polling client threads on one acceptor core, making the tail a scheduler-timeslice measurement. Switching that run to a multiplexed acceptor reduced kernel p99 from 81 002 to 207 µs and p99.9 from 112 001 to 1 819 µs. These are historical observations, not the current matrix's recipe or measurements.

The current load harness uses --multiplexed for Java-pure mono, bounded multiplexer workers for pool, and multiplexed shards for shard. The old 32-threads-on-one-core diagnosis does not describe the current full-recipe v4 captures; each retains its own delivery, stability and placement qualifications.

TCPDirect after the muxer

cpp/tcpdirect was median 78.77 µs at 17.6× spread with 30/32 completing. Giving the acceptor a real readiness signal via zf_muxer took it to 7.02 µs, 1.0×, 32/32. Raising rx_ring_max 8× first moved it 78.77 → 79.61 µs, which is how the buffer explanation was ruled out.

Every bypass row is native on both ends

A row labelled tcpdirect now runs TCPDirect at the acceptor as well as the initiator. Three cells previously paired a bypass client with an Onload, VMA-preload or plain-TCP acceptor, so the bypass acceptor was never on the measured path.

Historical Java/VMA results before offload verification

A plain Java ping-pong with no engine in it: kernel 17.18 µs, onload 4.08 µs (4.2×), vma 17.74 µs (−3%). A 150-line PoC then varied the one thing the engine cannot — NIO against blocking sockets, which take different syscall paths:

NIO 13.82 → 15.66 under VMA (13% worse); blocking 21.37 → 21.54 (1% worse). VMA_THREAD_MODE 0/1/3 changes nothing (14.30 / 14.32 / 15.32). VMA works on this box — C++ goes 22.18 → 6.77, a 3.3× gain on the same NIC — and the same JVM takes 4.2× from Onload. Those historical tests did not establish per-socket offload and do not show that VMA cannot accelerate Java. A later baseline exposed IPv4-mapped IPv6 sockets taking the kernel fallback. After explicit IPv4 selection and ownership repairs, identity-checked VMA statistics confirmed increasing RX/TX offload at all 64 endpoints. The old timings remain historical observations; socket-family evidence was not captured for each of those earlier PoC runs.

The C++ acceptor is not the reason — measured, not assumed

C++ is the fastest flavour at one session and among the slowest at 32. The obvious explanation was that the C++ bench drives CoreAcceptor (483 lines) while java-jni and dotnet-native drive fix::SocketAcceptor (1169), and only the second has had the concurrency work. That explanation was published here, then tested by building the missing arm (--acceptor-impl runner) and running both acceptors against the same clients on the same cores, alternating.

It is wrong. Medians are within noise (76.6 against 79.1 over three reps, the runner winning one), and at p99 the runner is worse in all three — 114.5 against 90.1, about 27%. The C++ row is already measured through the better of the two C++ acceptors, so switching it would make the cell look worse, not better. Whatever costs C++ its concurrency headroom is elsewhere.

Measured over three sittings through the same driver, the degradation from 1 to 32 sessions is cpp +14.9µs, java-jni +10.9, dotnet-native +9.6 — consistent in direction every time. The absolute ordering is not: run-to-run spread on these cells is 6–10µs against gaps of 3–7, so cpp came out slowest, then middle, then slowest again. Read the degradation, not the ranking.

A second caveat from the same investigation: the shard harnesses at M=1 measure about twice the latency of this driver on the same cell (cpp 65.95 against 37.21, java-jni 48.74 against 39.33). M=1 controls are therefore valid against their own M=2 cells and must not be read against this table.

A mono-thread cell that was three times too slow

dotnet-native/tcpdirect read 17.87µs mono-thread against 5.05 for a single session — a 3.5× penalty for concurrency that no other TCPDirect row shows. Re-measured at M=1 by the shard harness it reads 5.05 / 7.61 / 11.13, identical to the single-session cell, with all 32 sessions completing. The old figure was wrong, not the transport.

It had stood unquestioned because it was the only number in its row and had no p99 beside it to contradict it. That is the argument for measuring M=1 alongside M=2 everywhere: the control is what makes a cell falsifiable.

TCPDirect shard results were a topology false positive

The old C++, JNI and .NET Native TCPDirect shard rows launched one listener with a reuse-port flag. That is still one acceptor, so session completion did not prove sharding. A direct two-listener probe returns EADDRINUSE from the second zftl_listen: TCPDirect exposes no equivalent of SO_REUSEPORT. Those rows were withdrawn; the current capability contract still marks them no shard. SocketXtreme also remains out: its listeners bind, but independent pollers disconnect clients while competing for the shared completion stream.

TCPDirect's max is a clamp, not a reading

This cell's max was left blank because 4 294 967.29µs looked like corruption. It is not: latency_u32() saturates at 0xffffffff nanoseconds, so the value means at least 4.29 seconds — a real sample, reported at the only bound the type can express. That historical diagnostic treated it as a lower bound, not an exact maximum. It is not a current numeric latency-at-load claim from a complete, unclamped capture.

M=1 and M=2 agree closely (7.04 against 7.02 at p50) and both complete 30 of 32 with the same clamped max, so neither the two missing sessions nor the multi-second sample has anything to do with sharding. Both belong to the TCPDirect arm at this session count, and neither is yet explained.

Historical VMA reuse-port connection failures

Four historical cells were reported over fewer sessions than were offered, and the number that connected is the finding rather than a footnote to it: java-jni/vma 21 of 32, dotnet-native/socketxtreme 12 of 32 at M=2 and 19 of 32 at M=1, and dotnet-pure/vma 26 of 32. The failures are Connection refused or a disconnect during warmup, so the sessions are absent rather than slow, and the quoted percentiles describe only the ones that got in.

dotnet-pure/vma isolates it cleanly, because its M=1 control is healthy: 32/32 at p50 12.90, p99 15.04. Add a second acceptor sharing the port and it becomes 26/32 with p99 28 ms and p99.9 100 ms. One acceptor is fine; two on one port is not. Three flavours, three bindings, one shape — VMA runs its own TCP stack, and SO_REUSEPORT balancing is a kernel facility that its offloaded listeners have no reason to honour between processes.

M=1 fails too, so this is not sharding: a single acceptor is not a shard. It is the VMA-backed acceptor path not sustaining this session count, which the kernel, Onload and TCPDirect arms all do at 32/32 on the same rig in the same sitting. Worth noting that the C++ SocketXtreme acceptor manages 32/32 — so this is specific to the provider bindings, not to SocketXtreme itself.

Always measure M=1 with the same harness

The .NET Native multi-thread cells were nearly published as a regression. Kernel at M=2 reads p50 48.30µs against a mono-thread neighbour of 27.27, which looks like sharding making things worse. Running M=1 — not a shard, so a control — through the same harness gives 51.07. Sharding is a small improvement (51.07 → 48.30); the gap to 27.27 is between two harnesses, not between one and two acceptors.

The two harnesses agree closely where the transport is fast — onload 6.14 against 5.89, vma 6.69 against 6.67 — and disagree only on kernel, the slowest arm, where clients hold their cores longest and the difference in how many cores they are given bites hardest. A cross-harness number is worth what its control says it is worth, and nothing more.

Historical JNI/VMA multi-process shard failures

The retired java-jni driver reached this shape as M acceptor processes sharing one listening port. Kernel and Onload take it cleanly — 32/32 sessions each, Onload at p50 7.27µs. VMA completes only 21 of 32, and the stragglers are not slow but absent, with a worst sample of 3.08 seconds. VMA implements its own TCP stack, and SO_REUSEPORT load balancing is a kernel facility; there is no reason to assume it is honoured across processes that each hold their own offloaded listener. The historical report included the numbers the 21 that connected produced, and the completion count stated, rather than a clean median that hides a third of the sessions never arriving.

The current JNI/VMA shard implementation instead keeps M listeners and M pinned native serve threads inside one JVM. The old 21/32 result does not describe that topology; neither it nor the old reuse-port hypothesis establishes the cause of a new failure.

All three were measured in one sitting with identical parameters, verified by check-bench-params.sh. The kernel control read 27.50 here against the 23.95 already in the column, so this column carries roughly 15% of harness spread between the two shard drivers — enough to compare transports within it, not enough to read a 15% difference as real.

SocketXtreme shard completion was a topology false positive

Adding --zerocopy fixed the original client startup failure, but the replacement harness still launched one listener with a reuse-port flag. Its 32/32 completion proved a healthy mono acceptor, not sharding. A real four-listener probe gets all four binds and all four affinity records, then disconnects every client during Logon: independent SocketXtreme pollers cannot safely own the shared completion stream. The native-backed SocketXtreme shard cells now read no shard until a multi-listener completion dispatcher exists.

The pool wins once the client stops moving

The multi-thread column in the table above is confounded: clients and server shared ten isolated cores, so every core given to an acceptor thread was taken from 32 client JVMs. Replacing them with one client process driving all 32 sessions from a 4-thread pool fixes that, and reverses the result — the server pool now beats a single multiplexer 2.2× on the median (100.97 against 219.40) and matches it at p99. The earlier "pooling loses" reading was an artefact of the measurement, not a property of the design.

Two caveats stated rather than buried. Offered load had to drop to 2000/session: four client threads cannot sustain 320k/s, and at that rate every arm reads in whole seconds of backlog. And thread-per-session collapses entirely here — 32 spinning threads on one core against a client that no longer matches them one-for-one — which is a different failure from the one the table above records for it, and is not yet explained.

The multi-thread column is confounded on this box

Clients and server share ten isolated cores, so every core given to an acceptor thread is taken from the client processes. The pure-Java pool shows it plainly: one multiplexer with 9 client cores reaches p99 207 µs; four threads with 5 client cores reaches 1 689. That is not the acceptor model, it is 32 client JVMs on five cores. The same wall appears in java-jni (M=4 wins the median and loses p99.9 by 350×) and in the earlier placement work.

These rows are honest measurements of this historical configuration and should not be read as a verdict on pooling. The current same-host recipe instead uses disjoint client/server cores and one client process per client core, multiplexing its local sessions.

Correction. An earlier version of this page said the rows above no longer reproduced and blamed the machine. That was wrong. They reproduce exactly — p50 26.78 published against 26.78 / 26.22 / 26.08 re-measured — once the published recipe is used. The failed re-measurement had been run at 5000/5000 paced to 10000/s instead of unpaced at 20000/20000, where the path goes cold between messages and the same code reads 31.7. A code regression, machine drift and background load were each advanced and believed before the recipe was checked. bench-run now stamps every result with its parameters and check-bench-params.sh refuses a matrix whose cells disagree.

Sharding helps only the kernel, and costs tail everywhere

Four shards against one, both at 32 sessions. kernel p50 33.52 → 23.50 (a real 30% gain, since the syscall-bound path saturates a core) but p99.9 47.50 → 947. onload 5.92 → 7.53 and vma 6.80 → 11.35, both worse throughout. The bypass transports had headroom on one core, so extra shards only add contention. A pool is an ordinary-socket tool, not a general user-space-stack one.

The shard defect that hid all this

Every shard bound the port twice: Start() calls Listen() itself, and the shard called Listen() first. With SO_REUSEPORT both binds succeed, so each shard left an orphaned listener nothing ever accepted from while the kernel hashed a share of connections to it — M=2 lost 14 of 32, M=4 lost 15, the half you would predict from twice as many listeners as acceptors. Removing the redundant Listen() gave 32/32 in the retested configuration, not proof that unsupported transport/model combinations work.

The multi-thread column is capacity, not a fix

The gate for building shards — a 32-session median above 2× the same transport's single-session p50 — is closed on all five transports (1.03×–1.60×). One thread is not the bottleneck at this session count.