# Mellanox mono stall: the receiving port drops frames of live bypass flows

SCAN, 2026-10-02/03. Closed 32-session mono cells on the Mellanox pair
(`mlxA`/`mlxB`, socketxtreme and VMA) fail in about a third of runs: some
sessions stop getting replies and the run ends with every client `FAIL`. The
C++ and pure-Rust engines both show it. This record narrows it to the
ConnectX-4 Lx itself. Updating the card's 2016 firmware to 14.32.1912 removed
it: none of 30 runs failed afterwards (see "After the update" below).

## Symptom

A stalled session's request has left its sender and its reply never arrives.
In the client's stall disclosure (51141707a) the engine holds nothing: no
unsent bytes, no unread bytes and nothing staged on the ring. The stalled
sessions of one client process stop together. Some stop at their first order,
others after up to 18,000 round trips, while sibling sessions in the same
process run on. Either direction can be lost.

## What it is not

- **Not the engines.** cpp and rust-pure stall alike, and nothing is held in
  either engine at the stall.
- **Not neighbour state.** Both rig neighbour entries are PERMANENT (536b4ca66)
  and the stall rate did not change.
- **Not buffer starvation.** The RDMA receive-queue drop counter
  (`/sys/class/infiniband/mlx5_*/ports/1/hw_counters/out_of_buffer`) reads 0
  on both ports since boot, and no `ethtool -S` discard, drop, error, pause,
  per-priority or steering counter moves.
- **Not the kernel path.** A kernel ping crosses the pair both ways at every
  stall (mlxcap4, 6 stalls).

## What the port does

Diagnostic `rust_pure` builds read every NIC counter on both ports at the stall,
with the ports otherwise quiet. They also ran a kernel ping each way and a
promiscuous tcpdump on both ports. The diagnostics run as root inside the
namespaces under `bench-run`; the code is on branch `s/claude-mlx-stall`
(ac482b95a).

1. **The lost frames reach the receiving port and stop there.** They are
   counted in `rx_packets_phy` and `rx_prio0_packets`, but not in
   `rx_vport_unicast_packets` or any other counter. One stalled hold: 22
   frames at the port, none at the vport. Another: all 13 of the client's
   retransmissions dropped at the server's port.
2. **Kernel frames on the same port pass at the same moment.** In one 0.4 s
   window the client's port accepted 6 ping frames and dropped 2 server data
   frames (256–511 bytes).
3. **Switching the port into promiscuous mode releases the stuck flows.** When
   tcpdump started at a stall, which makes the driver reprogram the port's
   receive filters, every waiting session got its reply. That held in every
   stalled client of mlxcap3, mlxcap4 and mlxcap5. tcpdump itself captured
   nothing: the released frames went to the bypass stack's flow, not to the
   kernel.
4. **Promiscuous mode by itself does not.** With the server holding both ports
   promiscuous for the whole run (mlxprom), socketxtreme still failed 5 of 10
   runs and VMA 1 of 10. The frames were still dropped before the vport, and a
   tcpdump at the stall, now without a mode change, released nothing.

So the card holds a receive-filter state that drops the frames of particular
established bypass flows, silently and before the vport, and a driver
reprogram of the port's receive filters clears it.

## Leading candidate: the firmware

| Host | Card | Firmware | Kernel, libvma | Mellanox closed mono |
|---|---|---|---|---|
| SCAN | ConnectX-4 Lx, PSID `MT_2420110034` | **14.17.1010** (2016) | 5.14.0-687.44.1.el9_8, 9.8.80 | rust-pure: 14 of 46 socketxtreme and 10 of 35 VMA runs failed in these trials (0–67% by sitting) |
| HP | ConnectX-4 Lx, PSID `DEL2420110034` | 14.20.1820 | the same | 0 failures in the 27 mono rows in its run directories (cpp, java-jni, dotnet-native, rust-pure) |

The firmware is the clearest difference, but HP also differs in CPU, cables
and timing, so this is a correlation. Firmware 14.17 also cannot answer the
driver's version query ("fw query isn't supported by the FW" in the kernel
log).

## The update (needs root)

1. Update SCAN's ConnectX-4 Lx to the current 14.x firmware for its PSID with
   `mstflint` (query first: `mstflint -d 03:00.0 q`), then reset the card or
   reboot.
2. Re-run the trial with the published binaries: closed mono, socketxtreme and
   VMA, ten runs each. Ten clean runs of each would be good evidence. Any stall
   sends the investigation back to the receive-filter state.
3. If it is fixed, re-measure the Mellanox mono cells (and anything else on that
   pair the update could move) before publishing again.

## After the update

The firmware was updated the same day: 14.32.1912 burned with `mstflint`,
activated by a live card reset at 11:50. The MACs, GUIDs and PSID are
unchanged, and the old image is saved on SCAN. The published binaries then ran
closed mono on the Mellanox pair, with nothing ingested:

| Cell (closed mono) | Failed runs before | Failed runs after | p50 / p99 us, published | p50 / p99 us, after |
|---|---|---|---|---|
| rust-pure socketxtreme | 14 of 46 | 0 of 10 | 99.66 / 123.88 | 96.8–107.8 / 110–127 |
| rust-pure VMA | 10 of 35 | 0 of 10 | 88.80 / 1374 | 86.8–89.1 / 1202–1407 |
| cpp socketxtreme | intermittent | 0 of 5 | 74.50 / 156.03 | 73.3–74.9 / 155–159 |
| cpp VMA | intermittent | 0 of 5 | 68.20 / 498.31 | 67.4–67.9 / 500–507 |

At the old rate, ten clean runs of one cell happen about 3% of the time, and
twenty about 0.1%. One earlier sitting also ran ten without a stall (mlxcap2),
but its diagnostic toggled promiscuous mode on the server's port during the
runs, which releases stuck flows. The firmware is the fix. The next full
sitting on the pair will confirm it.

The closed mono latencies did not move: every figure is within the spread
between sittings.

HP was updated the same day to Dell's 14.32.2004 for its PSID
(`DEL2420110034`), from Dell package XGP2X, with a live card reset. Its MACs,
GUIDs and PSID are unchanged.

## Re-measured rows (SCAN)

SCAN's Mellanox cells were re-measured on the new firmware the same day: 30
closed and 24 paced rows (single, mono and pool), and 27 open v4 cells. HP's
Mellanox rows were not re-measured and still come from 14.20.1820.

Apart from one binary, everything was already installed, so the firmware was
the only change. The exception is `bench_roundtrip_vma_sx`, rebuilt from the same
backport branch as the installed `bench_roundtrip` (84ddb9ad1). Since 314c70937
the open driver refuses a C++ log without the tuning stamp, and the 2026-09-12
build had none. The new build adds the stamp and the pool change, which does
nothing outside Onload. cpp socketxtreme's closed and paced rows were measured
with it too, so all of its rows come from one binary.

- **Every cell completed.** None is published as a failure any more, and
  closed mono needed no retries. Open: cpp/socketxtreme/mono and pool went
  from partial to ok, and java-pure/vma/pool from fail to partial (now measured,
  p50 16.6 us). Paced: java-jni/vma/pool went from fail to partial. Closed:
  java-jni/vma/mono went from partial to ok.
- **The multi-millisecond maxima are mostly gone.** Closed java-jni/vma/mono
  3,101 to 148 us, closed cpp/socketxtreme/pool 7,230 to 3,108 us, paced
  java-pure/vma/mono 8,813 to 549 us, open java-pure/vma/mono 3,209 to 626 us.
  One went the other way: open dotnet-native/vma/mono 43 to 247 us, a single
  spike.
- **p50 and p99 mostly within the spread between sittings.** The exceptions
  are open rust-pure on VMA, 0.7 to 1.0 us slower at p50: single 7.79 to 8.80
  (8.46 in a repeat that was not ingested), and mono 9.77 to 10.67. Open
  java-pure/vma/mono rose 16.6 to 18.2 us. Their closed rows did not move.
  This is spread between sittings, not a firmware step. On the new firmware,
  ten collection-only sittings of open rust-pure/vma/single on SCAN read 7.75
  to 8.80 us at p50 (median about 8.2). Every run within a sitting sits at
  that sitting's level. cpp/vma/single, alternated with it, read 6.87 to 7.07,
  with one 7.49. The single pre-update sitting (7.79) cannot say whether the
  spread is new. What the spread is not:
  - **Not the binary.** On HP (Dell 14.32.2004), rust-pure/vma/single stayed
    within 7.06 to 7.19 over ten sittings, and SCAN's own `rust_pure`, swapped
    in on HP, held 6.62 to 6.70 over four.
  - **Not memory placement at launch.** rust-pure's in-process round trip over
    20 launches on SCAN gave a single cluster (2,987 to 3,068 ns).
  - **Not huge pages.** Fast and slow sittings log the same VMA warnings,
    with no huge-page fallback.
  - **Not the harness or the host as a whole.** The recipe, arguments and
    cores are identical, and rust-pure on the Solarflare card did not move.

  What remains is how rust-pure's loop meets VMA on SCAN's hardware: its CPU,
  or this NVIDIA-branded card and its firmware. Going further needs root, for
  a `perf` profile or VMA's own counters. vma_stats has to attach to a
  process's statistics file before VMA fills it, and copies of the file come
  out empty. Side finding: HP's installed `rust_pure` (b7b947e1) is about
  0.4 us slower on HP's rig than SCAN's build (0c1586ae), and HP's published
  rust-pure rows come from it.
- **Verdicts that moved against the numbers:** closed rust-pure and java-jni
  socketxtreme mono went from ok to partial on run-to-run spread, with every
  session complete.

**Three harness faults on main, none from the firmware.**

- 0d1068180 added the recipe field `rig_config_sha256`, which the open
  driver's pinning audit rejected, so every open sitting failed at startup.
  Fixed by e707e6b1d.
- 8d0c5867c made `gen-matrix.py` require `--data-dir`, so the open driver
  crashed at its first page render. Fixed by 1b47acc52, which passes
  `docs/bench` as both the data and the output root.
- One open sitting, dotnet-native/socketxtreme/mono, was aborted on its first
  attempt by a .NET Native client's open-loop correlation check ("Request ID
  aliases an outstanding request"). No frames were lost, and the retry passed.

## Evidence

On SCAN, under `~/libhft-bench-runs/claude-onload-pool-fix-20261002/`:
`mlxnet_*` (counters at the stall), `mlxcap2_*` (in-process capture, no
stall), `mlxcap3_*` (tcpdump at the stall), `mlxcap4_*` (kernel ping each way),
`mlxcap5_*` (every counter on both ports) and `mlxprom_*` (ports promiscuous
all run). Trial log lines are in `status` in the same directory.

The post-update runs are in `~/libhft-bench-runs/claude-mlx-fw-20261003/`.
The re-measure is in `~/libhft-bench-runs/claude-mlx-fw-remeasure-20261003/`: `closed_*` and
`paced_*`, `*_cpp_socketxtreme_stamped`, `open2/` (the published open sittings, `-p2`), the
failed attempts (`open/`, `open2-failed-gen-matrix/`), the repeat
`check_open_rust-pure_vma_single`, the control `check_open_rust-pure_onload_single`, the two
later rust-pure/cpp pairs in `vmastats/` and `vmastats/copies/`, and the replaced binary in
`previous/`. The spread study is in `~/libhft-bench-runs/claude-vma-states-20261004/` on SCAN
(the series and `inprocess-20-launches.txt`) and on HP (the series and `binary-swap/`).