# hp C++ SocketXtreme pool: lock-free completion hand-off

Recorded 2026-09-21 on hp. This follows the 100k investigation in
[`RECORD_2026-09-20_hp-socketxtreme-pool-100k.md`](RECORD_2026-09-20_hp-socketxtreme-pool-100k.md).
It is an engineering and reproducibility record, not a new public matrix row.

## Outcome

The C++ SocketXtreme pool had two independent shared-path costs:

1. VMA's completion/free/send API needs one caller for the listener-owned ring.
   Replacing the contended sleeping mutex around that API with the pool-local FIFO
   ticket lock removed the millisecond-scale queue collapse at 100k/s.
2. Even after that fix, each completion routed from the coordinator to its selected
   worker took the child's `std::mutex`.  That hand-off is structurally one producer
   (the coordinator) and one consumer (the worker), so it is now a bounded SPSC
   descriptor queue instead.

The SPSC change preserves the VMA API exclusion.  `PollAndRoute()`, packet free and
reply send remain inside `VmaPoolApiLock`; only queue publication/consumption no
longer takes a kernel mutex.

At the canonical 50,000-message/s, 32-session, three-pass open recipe, the two
profile-free controls changed from **13.17–13.26 us p50 / 29.78–30.09 us p99**
with the ticket lock alone to **7.55–7.74 us p50 / 19.90–23.94 us p99** with the
SPSC hand-off. Every one of the 4,500,000 measured replies in each capture was
recorded and the requested rate was sustained.

## Exact topology and artifact

| Item | Value |
| --- | --- |
| Host | hp, Xeon Gold 6254; profile-free C++ build |
| Transport | VMA 9.8.80 SocketXtreme, `VMA_RING_ALLOCATION_LOGIC_RX=0` |
| Placement | coordinator 8; workers 9-12; VMA internal thread 13; clients 14-17 |
| Shape | one listener/ring, four workers, 32 sessions, four client processes |
| Load | cell-v4 open, 50,000 aggregate messages/s, 5s warmup + 30s measured, three runs |
| Final installed binary SHA-256 | `367af070c5caff8af7724a852b3ee9a0e38c22eef61fa3b1d82f4027e64bf6f5` |
| Compile-time profile stamp | `tuned_build=0;profile_fingerprint=none` |

The 100k ticket-lock controls are retained in:

```text
/home/yann/libhft-bench-runs/socketxtreme-pool-100k-ticketlock-20260921T000958Z
/home/yann/libhft-bench-runs/socketxtreme-pool-100k-ticketlock-repro2-20260921T001501Z
```

They both completed 3,000,000 measured replies in one 30-second pass, at
20.59/39.60/43.95 us and 20.61/39.73/43.89 us (p50/p99/p99.9).

## Implementation contract

`FixSocketVma::EnableCrossThreadPump()` is opt-in and is used only after a
listener-owned ring accepts a fixed pool child. The coordinator is its sole
producer; that child's pinned worker is its sole consumer.

- The producer first checks that every child has room for a full
  `socketxtreme_poll()` batch (16 descriptors). It therefore never owns a VMA
  completion it cannot publish.
- Producer descriptor write + release-store of tail synchronizes with the
  consumer acquire-load of tail. The consumer copies the descriptor before its
  release-store of head.
- A copied packet descriptor is held in the worker's `m_heldPacket` until
  `ConsumeZc()` returns the VMA buffer. Reusing the SPSC descriptor slot cannot
  invalidate that VMA borrow.
- `Close()` drains every queued packet claim before discarding the socket. This
  matters because a positive VMA poll transfers ownership even if the worker
  stops before its next `PeekZc()`.
- Ordinary SocketXtreme client, single and mono paths do not enable this queue.
  They retain the established retained-batch path and acquire no new atomics.

The queue capacity remains 256 completions, with the existing 16-completion
admission reserve. Its producer and consumer indexes occupy separate cache lines
without imposing an over-aligned allocation requirement on the C++11 socket type.

## Verification

```bash
make -C engine/cpp fix-vma-correctness
make -C engine/cpp vma_session vma_session_count
```

The VMA ownership suite passes its existing poll/accept/failure coverage plus:

- 8,192 cross-thread packet hand-offs, forcing repeated SPSC wraparound while
  respecting the real full-poll-batch admission rule;
- ordered payload and exact-once VMA buffer release at every hand-off; and
- close with routed but unread packets, proving all queued VMA claims are freed.

The two full raw captures from the final artifact are:

| Capture | p50 / p99 / p99.9 / max (us) | Delivery |
| --- | --- | --- |
| `socketxtreme-pool-50k-spsc-ticketlock-20260921T014310Z` | 7.55 / 23.94 / 24.94 / 32.89 | 4,500,000 / 4,500,000 |
| `socketxtreme-pool-50k-spsc-ticketlock-repro-20260921T014815Z` | 7.74 / 19.90 / 21.17 / 31.84 | 4,500,000 / 4,500,000 |

Both captures attest coordinator, worker, client and VMA-internal-thread
placement; all workers reached eight sessions; all raw files are complete; and
there is no poll failure, VMA positive-batch refusal, marker stall, send refusal
or missing reply.

## Why the row is still partial

`PARTIAL` here is a strict stability classification, not an incomplete run. The
client result records are `status=pass`, have zero deficits and report 100% of
their requested per-session rate. The aggregate row takes the worst client
percentile, and two flows still have run-to-run p99 variation. In the second
SPSC capture, for example, client 2/session 4 reported run p99 values of
23.99, 10.93 and 24.43 us; client 3/session 5 reported 23.67, 12.19 and
23.84 us. That exceeds the benchmark's 25% failure CV threshold even though
all work completed.

This residual is materially smaller than the pre-SPSC tail but is not explained
away as a harness issue. The connection-to-worker admission log, raw samples and
per-run output are retained for a future flow/RX-queue correlation experiment.
Do not replace the qualified historical/public matrix cell from these controls
until a full same-recipe matrix refresh meets the stability gate.