# HP llfix receive buffers, sending rates and OS control

2026-10-08. Continues the [TCP tail investigation](RECORD_2026-10-07_llfix-hp-tcp-tail.md)
with the [qualified receiver/rate preparation](RECORD_2026-10-07_llfix-hp-load-preparation.md).
The receiver/rate capture, independent cold audit and complete evidence
verification on both HP and BEELINK2 pass. OS control is prepared and awaits
the separate root operator step.
These are observed diagnostics, with no quiet timing, vendor reproduction,
Stage 2 ranking or release admission.

## Receiver buffer matrix

Two fresh-process runs per cell, forward then reverse, use the original
unmodified llfix 1.0.7 TX benchmark and 1000000 measured sends plus 1000 warmups.
Receiver spin remains enabled. The controller and observers use CPUs 0-7;
native client/peer remain on CPUs 17/16 and the physical SFC9220 DAC in separate
namespaces, with both Onload TCP loopback shortcuts disabled.

Values below are medians of per-run statistics, not pooled quantiles. The
maximum column is the actual largest native-reported maximum across both runs.

| Receiver control | Actual receive buffer bytes | p50 ns | p95 ns | p99 us | Mean ns | Largest maximum ms |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Default, spin | 131400 | 644 | 2245 | 6.9135 | 1945.208 | 49.2159 |
| Request 262144 bytes, spin | 525600 | 646 | 2163.5 | 2.7420 | 1937.399 | 46.4332 |
| Request 1048576 bytes, spin | 2098020 | 642.5 | 2160 | 2.6205 | 1950.344 | 55.7013 |

The larger receive buffer closes the previously observed p99 numerical gap:
individual large-buffer runs report 2607 and 2634 ns, versus the
[website's 2663 ns](https://www.llfix.net/), checked 2026-10-08. Median p99
falls 62.1% against this sweep's default buffer. Its p50 remains 12.13% above
the vendor's 573 ns and p95 remains 36.02% above 1588 ns. This result identifies
a useful receive-buffer control; it does not establish matching distributions
or a reproduction of the vendor hardware, live topology or stack settings.

Mean remains approximately 1.94–1.95 us; the rare tens-of-milliseconds stalls
remain. Sender snapshots still show about 0.63 MB of queued unsent payload and
hundreds of thousands of receive-window stops. The larger buffer raises the
minimum observed sender window from below one MSS to approximately 8 KB.
These are sampled aggregate observations, not per-call causality or delivery
times. Successful local SDK return does not establish individual NIC enqueue,
peer delivery or acknowledgement latency. Cold bilateral journals separately
check that every order eventually arrives with its required values and owner.

The vendor's
[pinned complete native output](https://github.com/CorewareLtd/fix-engine-benchmarks/blob/165cc999fa283a4ead17a7b935c5d3f850fec851/fix_client_benchmarks/llfix/results.txt)
also reports mean 6009.729945 ns and maximum 348.331 ms. A matching p99 alone
therefore does not describe the full tail on either host.

## Paced sender matrix

This uses the separately qualified paced sender, preserving the entire timed
TX task byte for byte and scheduling outside the SDK service-time probe. It
keeps the default receiver buffer and spin enabled. Requested rate is a target,
and achieved rate includes measured runner/pacing time. Scheduling re-anchors
missed deadlines without burst catchup; this remains a service-time model with
retained scheduler delays, not a coordinated-omission-free arrival benchmark.

| Target calls/s | Achieved calls/s, two-run range | Median p50 ns | Median p99 us |
| ---: | ---: | ---: | ---: |
| Unpaced | 506245–507790 | 678 | 6.4285 |
| 100000 | 98202–98635 | 1454 | 3.7070 |
| 250000 | 238188–239529 | 1452.5 | 3.3475 |
| 500000 | 434442–435221 | 1461.5 | 3.7020 |
| 750000 | 505633–507267 | 680.5 | 4.7930 |

The highest target saturates near 506000 calls/s. Lower pacing changes the
service-time distribution rather than uniformly improving it. Original and
paced sender results retain their distinct runner identities. The separate
summary retains full per-run primary/delay statistics and actual maxima.

At 100000–500000 targets, sender snapshots show zero queued unsent payload;
unpaced and 750000-target snapshots show about 0.63 MB. Lower-rate p50 is
nevertheless higher. Local send cost varies with the TCP workload shape, so
a smaller local p50 does not establish earlier NIC or peer delivery. There is
no per-call attribution of those aggregate state observations.

Median-of-run scheduler p99 is 105, 105.5, 1983.5 and 3631 ns at the four
nonzero targets. Actual maximum retained scheduler delays range from 30.308 to
50.084 ms. Re-anchoring and separate retained delay series are explicit; missed
virtual arrivals are not reconstructed or folded into local service samples.

## Existing OS policy and next control

After capture, `tuned-adm active` reports `latency-performance`, with its daemon
active since the 2026-09-18 boot. The actual profile file is retained in a
separate read-only post-capture receipt; it requests performance governor,
min_perf_pct=100 and `force_latency=cstate.id_no_zero:1|3`. This discovery is
separate from the pre/post-capture CPU snapshots.

HP already boots with `isolcpus=domain,managed_irq,8-17`, `nohz_full=8-17`,
`rcu_nocbs=8-17` and `irqaffinity=0-7`. Benchmark CPUs have no SMT siblings;
performance governor and performance energy preference are selected, with
intel_pstate min/max performance percentages both 100. NUMA balancing is off.
The watchdog mask is 0-7 and benchmark CPU NMI counters do not increase across
the capture interval. C6/C1E usage and residency counters likewise do not
increase; C1 usage does. These snapshots include setup and gaps between pairs,
and do not correlate individual idle exits or interrupts to timed SDK calls.

Turbo is disabled. `cpupower frequency-info` reports boost unsupported/inactive
and hardware limits 1.20–3.10 GHz; intel_pstate reports turbo_pct=0. This does
not identify the firmware setting responsible. No BIOS, reboot, turbo policy,
IRQ service, boot argument, system-wide TCP sysctl or persistent OS setting is
changed by this generation.

The [new private OS control](../../bench/llfix_reference/diagnostics/os_controls/README.md)
compares baseline → a temporary zero microsecond idle-exit latency request →
baseline. Each arm runs original-spin, original-rx16 and rate-100000 twice in
balanced order. The root parent opens `/dev/cpu_dma_latency` only after its
unprivileged worker verifies the prepared source/runtime and both high-priority
host CPU leases. It checks effective zero and closes the request before the
next baseline and before lease release. Process death also closes the
descriptor and the kernel removes that request automatically. The constraint
is global for its short active interval; no permanent policy or permissions
are changed. The
[kernel PM QoS description](https://www.kernel.org/doc/html/latest/admin-guide/pm/cpuidle.html)
explains these descriptor and idle-governor semantics.

The operator wrapper is `~/run-llfix-os-controls.sh` on HP. Automatic `sudo -n`
is refused because the existing password-free policy covers protected namespace
launchers and does not grant this PM QoS operation. It requires the user's
manual password entry. At this checkpoint no OS-control native run is claimed.
The script queues automatically; cold audits follow separately at shared/high.

## Source and verification

The installed receiver/rate generation verifies exact root ownership, modes
and hashes at `/usr/local/libexec/llfix-reference-hp-load-v1-7fc38f148858` plus
its four fixed namespace launchers. User installation log
`install.n6TDEBoe.log` is retained. Frozen 60-file source and unchanged hft-bench
headers remain pinned at `9338947d1da982a9c762bbc55a97361f4bf9f6a8`, GCC 11.5
and the previous source-built Onload inputs. Installed original TX/peer hashes
and all older payloads, SDKs, helpers, Stage 2 campaigns and raw evidence remain
unchanged.

The first cold audit retains a failed receipt after the default cell passes:
Onload renders changed options as `EF_TCP_RCVBUF: 262144 (default: 0)`, while
the frozen regex accepts only a bare number. The exact value and captured
native environment already agree. A separate read-only continuation accepts
only that exact default-zero annotation. It checks the frozen controller SHA256
`1a40c2f02420a5cc601dcf2ab79685d6bdec9ed350baface5988d09116c94faf`
before execution. Reversing three AST constant changes restores its complete
original audit AST: one display regex and two distinct output filename
references. All field/owner/count/raw checks, source/runtime guards and grades
are retained; the original failed audit and native inputs are untouched.

OS and continuation source is preserved at `7a9473165` on
`s/codex-llfix-reference-os-controls-source-20261008`, integrated into main as
`b1a1b5fe3`. Twelve boundary checks pass locally, zero skips. The staged
65-file capsule includes the unchanged previous 60 files and five new private
files. All twelve also pass on HP at shared/high, zero skips. Real CLI help,
unprivileged parent refusal and missing-lease worker refusal pass without native
execution or runtime directory creation. Actual root/device controls remain
pending and separately graded.

All sixteen receiver/rate pairs return zero for both native processes. The
corrected cold continuation passes every original audit check on 32032000
persisted endpoint order records, including warmups and both copies. Total
measured client calls are 16000000; paced raw primary and scheduler samples are
10000000 and 8000000. Counts, required converted FIX values/groups, envelopes,
checksums and contiguous numeric owners are checked independently. Paced CSV
quantiles/means/true maxima, binary series, clocks, achieved rates and clamp
disclosure are also independently recomputed. No native rerun or original
receipt alteration is needed for the display-parser correction.

OS cache: `~/.cache/libhft/llfix-reference-hp-os-v1-bb98dba4ec1d`.
Controller SHA256:
`bb98dba4ec1d10e8b135e54002b89ff4abd482211ab3de07139c7ea427aacccd`.
Home wrapper SHA256:
`c5b9ac03fa29b8473559612d476cf2211191f2f456e418b486a9c19fe81a7d42`.

## Retention

The [audited summary](evidence/llfix-hp-load-os-20261008/metadata/summary1.json)
retains every run and the effective buffers, TCP observations, raw-series
statistics, rates and actual maxima. The
[metadata selection](evidence/llfix-hp-load-os-20261008/selection-manifest.json)
contains 77 exact sealed files plus the seal, replica receipts and a separately
hashed post-capture tuned-profile supplement. Full journals, raw CSV/HFTBRW,
all native binaries, complete source/input capsules and operational scripts
remain in the complete archive.

HP and BEELINK2 independently stream all 676 sealed files, checking every
member's SHA256, size and mode. Their passing receipts match exactly. The seal
covers 27131608480 raw bytes; the archive is 588707668 bytes. Both hosts retain
`~/archive/libhft/codex-llfix-reference-20261008/hp-load-results1.tar.gz`.
Archive SHA256:
`5a99f3ac7a1dc580da346db7680341aa9f6250da33cce1dd92c78c9ae10b78d5`.
Seal SHA256:
`260fd6204557d66aa94757101d150d901dbc6a878bb9c97984fd8df65043d311`.
OS prepared caches and the root operator wrapper remain active inputs. Cleanup
removes only the exact completed receiver/rate build runtime and its disposable
materialization after both replica receipts match. Its source/shape guards and
native PID/start-time checks pass. Uncompressed primary evidence, both complete
archives, source caches, installed payloads and the future OS runtime directory
remain intact. See the [cleanup receipt](evidence/llfix-hp-load-os-20261008/cleanup1.json).
The [integration receipt](evidence/llfix-hp-load-os-20261008/integration1.json)
verifies all 60 original source files, all five new OS/continuation files and
all 77 selected metadata files unchanged after integration. Twelve focused
checks pass, zero skips, and all local documentation links resolve. A WIP
claim conflict is resolved by preserving the concurrent flow/owner review
claim and removing only this completed evidence claim. Concurrent Rust changes
remain in main; no benchmark code or captured generation changes on rebase.