# Onload pool: one stack lock shared by every worker

SCAN, 2026-10-02. The closed 32-session onload pool cell stalled for most of a
second about once a run and had a median three to six times what the stack can
do. Both come from the pool's workers sharing the listener's Onload stack.
The fix is 59f44ec3e; the rows are b1227341d, 7c75935a0 and 275443b57.

## Symptom

rust-pure's closed onload pool row carried a 285–578 ms maximum. The reported
maximum is the per-run worst averaged over three runs, so that is one round trip
of about 0.85 s. Raw samples (`BENCH_RAW_SAMPLES=1`) placed it within the first
few milliseconds of a measured run, on the eight sessions of one pool worker,
ending at the same instant on all eight. The other 24 sessions kept their
ordinary 100–200 us worst case throughout. A control run of the C++ server
stalled the same way (352 ms reported, against a published 2.8 ms).

## What it was not

- **Not the engine's read path.** Timing the worker's calls named the reply's
  `send()`; receive and flush never exceeded 5 ms.
- **Not stack-lock churn from empty reads.** Switching the worker to `poll(0)`
  readiness left the stall in place and cost 15 us of median.
- **Not memory fragmentation.** Free high-order pages were plentiful.
- **Not a stale neighbour entry.** Onload's "ARP confirmations (tried)" counter
  moved 0 to 1 in one stalled run and stayed 0 in one clean run, and 536b4ca66
  pinned the rig's neighbour entries on that reading. With both entries
  PERMANENT the stall was unchanged (4 of 4 runs) and the counter stayed 0.
  Two runs were a coincidence; 94bc9fd0a corrects the script's comment. The pin
  stays as hygiene.

## Cause

A watchdog thread in a diagnostic server printed the kernel stack of a worker
that had been inside one `send()` for 20 ms. Three stalls, the same stack:

    oo_eplock_lock <- oo_eplock_lock_rsop <- oo_fop_unlocked_ioctl <- ioctl

That is the Onload **stack lock**. The chain, each link read from the Onload
source built on this host and counted by `onload_stackdump lots`:

1. A non-blocking TCP send takes its packet buffer from a lock-free pool
   (`ci_netif_pkt_alloc_nonb`, `tcp_send.c`). When the pool is empty it needs
   the stack lock, and if `trylock` fails with nothing yet filled it takes the
   blocking lock. `tcp_send_nonb_pool_empty` and `tcp_send_ni_lock_contends`
   both read 47–50 per run: every new peak of replies in flight, such as the
   first moment all 32 sessions send together.
2. The latency profile sets `EF_POLL_USEC=100000`, and `netif_init.c` derives
   `buzz_usec = min(spin_usec, 100)` from it ("Don't buzz for too long by
   default"). After 100 us of spinning the thread sleeps in the kernel.
3. The lock has no hand-off (`oo_eplock_lock`, `eplock_resource_manager.c`): a
   woken waiter re-reads the lock word and sleeps again if it is held. A
   spinning worker that has just released the lock re-takes it within a
   fraction of a microsecond, long before the sleeper is back on its core.
   `lock_wakes` read about 37,000 for one 0.85 s stall. Per-core counters show
   the same thing: the stalled worker's core at 0 user cycles and about 14%
   kernel, the other workers' cores with a little kernel time each.
4. `deferred_work` read about one million per run: nearly every reply failed to
   get the lock itself and was handed to whichever worker held it. That, not the
   stall, sets the cell's median.

It is a property of the pool topology under Onload: an accepted socket lives in
the listener's stack, so M spinning workers drive one stack. The shard cell does
not have it, because Onload clustering gives each `SO_REUSEPORT` listener its own
stack.

## Fix

Two changes, each measured on its own with a diagnostic build before anything
was published (five runs each, closed onload pool, rust-pure):

| Variant | Stalls | p50 us | p99 us | max |
|---|---|---|---|---|
| shared stack, as published | 4 of 4 | 43 | 96–140 | 280–578 ms |
| `EF_BUZZ_USEC=100000` | 0 of 5 | 43–46 | 125–142 | 157–225 us |
| `EF_BUZZ_USEC=-1` | 0 of 5 | 43–45 | 138–142 | 172–210 us |
| a stack per worker | 0 of 5 | 14.6–15.6 | 25–28 | 67–88 us |

- **A stack per worker.** Each worker calls `onload_set_stackname` at thread
  scope and `onload_move_fd` on every socket it adopts, before the first read.
  Onload moves a socket only before its first read or write; all 32 moved and
  none was refused. In the C++ pool (`fix::AcceptorThreadPool`, used by the C++,
  JNI and .NET Native servers) and in the pure-Rust pool
  (`net::OnloadThreadStack`). Both look the symbols up in the preloaded library
  and do nothing without Onload; `LIBHFT_ONLOAD_WORKER_STACKS=0` keeps the shared
  stack. A worker that claims a stack prints `[pool-onload] worker N
  own-stack=yes`.
- **A longer lock spin.** `bench-run` exports `EF_BUZZ_USEC=100000` for every
  Onload process. It removes the stall and leaves the median where it was. It is
  there for the pure-Java and pure-.NET pools, which cannot place a socket on a
  per-thread stack and still share one.

## Published rows

Onload pool, p50 in us, before and after:

| Flavour | Closed | Paced | Open v4 service |
|---|---|---|---|
| cpp | 62.84 → 10.29 | 9.30 → 6.98 | 9.14 → 7.40 |
| rust-pure | 43.11 → 13.48 | not implemented | 10.63 → 8.88 |
| java-jni | 59.07 → 15.96 | 10.93 → 9.19 | 11.74 → 9.77 |
| dotnet-native | 57.50 → 20.97 | 11.60 → 9.22 | 14.19 → 11.31 |
| java-pure | 44.32 → 43.84 | 18.57 → 17.04 | 17.40 → 18.51 |
| dotnet-pure | 42.56 → 42.51 | 18.01 → 18.31 | 18.01 → 18.18 |

Three verdicts moved against the numbers: java-jni and dotnet-native closed went
from ok to partial, and dotnet-native open from complete to latency-variation,
while every latency figure in those rows improved.

## Binaries

The C++, JNI and .NET Native bench binaries on SCAN are **not** builds of main.
They are built from branch `s/claude-pool-backport`: 010d929ba, the source the
2026-09-12 installs were built from, plus the pool change (c9d3b3649) and, for
`bench_roundtrip` only, the C++ tuning stamp (84ddb9ad1). The Java and .NET
managed code installed on 2026-09-12 has to keep matching its native library,
which a build of today's main would not after the API renames, and every row
not re-measured here stays valid because the change is inert outside an Onload
pool. Rebuilding any of the three from main means rebuilding and re-measuring
those flavours entirely.

The stamp is needed because the open-matrix driver has refused a C++ cell whose
logs carry no `bench_tuned_profile=` line since 314c70937 (2026-09-21). Closed
and paced runs do not check it, so a build of older source fails in open mode
only, as "startup affinity audit failed".

`rust_pure` is main at 59f44ec3e, untuned. The tuned rows of 1a16fec23 were
replaced: no other row on the page is tuned, the recipe's `tuned_profile` stamp
cannot record a Rust profile, and the two builds differ by under 0.1 us at one
session.

## Left open

- The pure-Java and pure-.NET pools still share one stack and keep a 43 us
  closed median.
- The Mellanox flow stall is unexplained. In this sweep socketxtreme mono and VMA
  mono each needed a second attempt. Pinning the neighbours did not change it.
- rust-pure closed onload mono read 90.6 us in the published sitting, 79.95 us in
  a repeat with the same installed binary, and 79.5 us in the morning's tuned
  sitting; the open cell does not move (9.99 against 10.01). It is variation
  between closed sittings, not the build.

## Evidence

On SCAN, under `~/libhft-bench-runs`:

- `claude-rust-pure-tuned-20261002/diag_pool_onload_*`: raw samples and per-core
  counters (`raw1`), call timing (`calltimes`), poll readiness (`pollready`),
  control-plane counters (`cplane`), kernel stacks (`watchdog`), Onload counters
  (`counters3`), the lock-spin runs (`buzzinf`, `buzz100ms`) and the per-worker
  prototype (`perthread`); `verify_pin_*` for the neighbour check.
- `claude-onload-pool-fix-20261002/`: `tip_*` and `backport_*` are the trials of
  the real builds before install; `sweep/` is the published sitting, with
  `previous/` holding what was installed before.