# HP llfix TCP tail investigation

2026-10-07. Follow-up to the [profile placement controls](RECORD_2026-10-07_llfix-hp-onload-profile-controls.md),
using the same installed original llfix 1.0.7 binaries, physical SFC9220 DAC,
client/peer CPUs 17/16 and frozen 39-helper generation. Both TCP loopback
shortcuts remain disabled. No SDK, Stage 2, full-session fixture, shared helper,
root payload or previous result changes.

Final cold audits, source-bound analysis and both complete archive replica checks pass.

## Results

The strongest demonstrated contributor remains receiver polling. A new balanced
four-pair run with user-space cycle profiling and live TCP observations gives
median-of-two p99 18.294 microseconds at default and 7.2505 microseconds with
receiver-only spin, a 60.37% reduction. Its p50 is 648.5 versus 645 ns. These
are instrumented diagnostic results, without quiet/timing/vendor admission.

Eight further fresh-process pairs keep the receiver spinning and omit perf
sampling; TCP/process observers remain active. Each pair measures 1,000,000
original TX calls after 1,000 warmups, with bilateral persistence enabled.
Balanced order is spin/large-buffer/small-buffer/receive-batch, then reverse.

| SDK control | Median p50 ns | Median p95 ns | Median p99 us | Median mean ns |
| --- | ---: | ---: | ---: | ---: |
| Default buffer, receiver spin | 649.5 | 2204 | 6.7855 | 1956.593 |
| Client TX buffer requested 4× larger | 672.5 | 2254 | 6.6375 | 1938.624 |
| Client TX buffer requested 16384 bytes | 1128.5 | 3326 | 20.5180 | 2083.089 |
| Receiver read batch 8192 → 32768 bytes | 646.5 | 2180 | 7.5090 | 1934.044 |

The larger-buffer p99 difference is only 2.18% in this two-repeat control. It
does not close the gap to the [website's 2.663-microsecond p99](https://www.llfix.net/).
The small-buffer p99 is 3.02× the same-series baseline; larger receive batches
do not improve it. No statistical confidence interval or pooled quantile is
inferred from two runs. Each root retains its actual per-run statistics,
including mean/maxima and the original first-CPU-MHz conversion scale.

## What the TCP state shows

During the profiled default/spin pairs, live sender snapshots show approximately
600 KB of queued **unsent** TCP payload, with the advertised receive window
often below one MSS. Receive-window stop counters grow above 900,000 while
congestion-window stop counters remain zero. Default receiver stacks show more
than 80,000 interrupts; spinning receiver stacks show zero interrupts. These
are aggregate observations over each pair, not counts of slow SDK calls.

The requested client buffer sizes take effect: observed sndbuf is 425984 bytes
at default, 1703936 bytes for the larger request and 32768 bytes for the smaller
request. The larger buffer permits approximately 2.5 MB of queued unsent payload
without materially improving p99. These queues can exceed nominal sndbuf:
Onload's state also exposes its packet-based send capacity. Requested versus
observed buffer sizes are retained independently; no sysctl was changed.

The pinned [Onload dump implementation](https://github.com/Xilinx-CNS/onload/blob/dde226d7/src/lib/transport/ip/tcp_debug.c)
defines queued send bytes as enqueued sequence minus transmitted sequence and
maps its rwnd/cwnd display to separate transmission-stop counters. Its
[transmit implementation](https://github.com/Xilinx-CNS/onload/blob/dde226d7/src/lib/transport/ip/tcp_tx.c)
increments the receive-window stop counter when a pending packet cannot fit.
These counters are neither stalled nanoseconds nor the number of failed SDK
sends. They confirm flow-control pressure, not a one-to-one cause for p99.

The four profiled pairs' observed stacks retain zero retransmissions, main-pool allocation failures,
memory-pressure entries and memory-pressure drops. The lowest observed free
packet counts remain above 31,000 in the profiled pairs, consistent with the
earlier packet-pool controls giving no improvement. Last observed client
nonblocking subpool-empty counters are nevertheless nonzero (22–40 in these
four profiles). That is retained separately from main-pool pressure, without
per-call attribution. The snapshots do not exclude every transient below
observation cadence.

The server's effective accepted-socket receive buffer is about 131400 bytes.
Although its settings parser and FixServer forward socket/quick-ACK options,
the pinned TcpReactor never consumes those accepted-socket options. A planned
quick-ACK setting would therefore be an ineffective control and was not run.
Its source and live-state evidence are retained; no vendor SDK patch is applied.

## Profiles, faults and measurement boundary

The four cycle profiles have actual samples for both native processes, no lost
samples and readable perf.data. Onload accounts for roughly 55–60% of sampled
client user-mode work in the initial comm-based summaries, with ci_tcp_sendmsg,
send and send-result handling among the largest self symbols. PID-filtered
cycle-weighted summaries give 55.65%, 56.42%, 57.35% and 60.29% Onload shares
and 2065–2089 client self samples per pair. Profiles include process
setup, SDK sends, statistics sorting and shutdown; kernel time is excluded.
They cannot assign sampled CPU fractions to individual p99 calls.

Client live process snapshots show about 69,000–93,000 additional minor faults
and zero major faults in these profiled runs. The pinned MemoryMappedFile
zero-writes missing file bytes, then maps without MAP_POPULATE. Client
persistence remains a candidate contributor to residual tails, but first-touch
faults are not correlated with measured calls. The earlier fixed-receiver
persistence-disabled control worsened p99, so fault counts alone do not establish
that removing journaling improves the overall send tail. Run-queue wait snapshots
also cannot account for time blocked in kernel/file operations.

The SDK returns the client socket to nonblocking mode and retries until the
complete message buffer is sent. Its local duration can depend on TCP progress.
However, queued unsent payload demonstrates that a successful return does not
establish NIC enqueue, remote delivery or acknowledgement time. The reported
test is useful for SDK construction/serialization/send cost under its transport
conditions; it does not directly measure wire-to-wire order latency.

The remaining best-cell p99 is about 2.49× the vendor report. Receiver polling
and actual TCP pressure explain important local effects, but do not establish
the cause of every remaining difference. Vendor live peer/topology/stack sharing,
hardware, compiler, Onload version and timestamp precision remain different or
unverified, as the prior record details. No vendor reproduction is claimed.

## Provenance and verification

Source bbdd65c46a97647329c10abd5daa7399a8a02ad5 owns the twelve completed
native captures/audits; 92a9a1d0d7d2a14aed67b1f59462b9da2c157029 adds the
source-bound analyser and two additional boundary controls. Eight then ten
focused checks pass locally; all ten also pass on HP with zero skips. Twelve
completed controls independently validate 24,024,000 persisted order records
across both endpoints. The failed profiler grading attempt's separate native
continuation validates another 2,002,000, without promoting its original failed
capture. Total measured client calls are 13,000,000; total cold-validated endpoint
order records are 26,026,000, including warmups and both copies. HP source
capsules verify exact bytes/modes,
including the unchanged sibling utility helper. Installed binaries, base and
three fixed profile launchers, original source/configuration headers and all
39 frozen helper inputs are hash checked before each capture/audit. HP jobs use
high priority CPU queues; live captures hold both shared and isolated pools.

An initial launch refused an output outside the installed runtime root before
native execution. The first native/perf attempt then finished both peers but
its controller rejected perf's SIGINT return after data flush. That capture
remains failed. The later controller accepts a signal exit only when it
requested the stop, and cold perf report must read native samples with zero
loss. A separate read-only continuation checks the failed attempt's native
journals and perf data without changing its original grade. GitHub's initial
server-side push errors were retried normally; source checkpoints and dated
preservation refs are pushed.

The combined audited summary completed, then an operational PID-profile parser
refused perf's padded semicolon fields. Its failed script/receipt and valid
original report are retained. A separate continuation strips field padding,
reuses the unchanged combined summary after exact recomputation, and reads the
same perf.data matching captured hashes into fresh PID-filtered reports. No native rerun or
old report alteration is used for that correction.

## Retention

The [audited summary](evidence/llfix-hp-tcp-tail-20261007/tcp-tail-summary1.json)
retains every per-run statistic and TCP/process series. The
[PID-filtered profiles](evidence/llfix-hp-tcp-tail-20261007/tcp-tail-operations1/pid-profile-summary2.json)
bind each extracted report to the captured native PID and perf.data hash. The
[metadata selection](evidence/llfix-hp-tcp-tail-20261007/selection-manifest.json)
contains 120 exact sealed metadata files plus the seal and two replica receipts.
All five qualified diagnostic source bytes/modes still match the integrated tree.

HP and BEELINK2 independently stream all 320 sealed files, checking every member's
SHA256, size and mode. Their passing receipts match exactly. The seal covers
21,939,211,650 raw bytes; the compressed archive is 387,468,699 bytes. Both hosts
retain `~/archive/libhft/codex-llfix-reference-20261007/hp-tcp-tail1.tar.gz`.
Raw journals, perf.data, supporting source copies/capsules and operational
scripts remain in that archive. Previous campaign seals and rankings are intact.

Archive SHA256:
`392895f729067828e3114447b8b0aa94e4e5a26e641d16e127c6013ecfdf2930`.
Seal SHA256:
`4a795fbe32c7a5800b2962b9853bd31ce2d03fb0e5bd03772096069e10d58d10`.

After both replica checks, the [separate cleanup receipt](evidence/llfix-hp-tcp-tail-20261007/hp-tcp-tail1-cleanup.json)
records removal of this session's HP build outputs and three disposable source
capsules. It first matched all 320 source files to the seal, checked cache copies
against their retained counterparts and confirmed recorded native identities
were absent. Its [exact script](evidence/llfix-hp-tcp-tail-20261007/hp-tcp-tail1-cleanup.py)
and receipt are retained outside the original seal. Installed launchers, original
cache inputs and all kept archives remain. The local worktree/build root is
closed after full publication; main holds the result and dated refs retain its
source generations.