# HP llfix Onload latency investigation

2026-10-06. This follows the [retained physical TX campaign](RECORD_2026-10-06_llfix-hp-onload.md).
The question is why HP's original-program Onload p50/p99 of 648/18,406 ns differs
from the vendor's 573/2,663 ns. The original campaign's rejected quiet grade,
installed generation 3 binaries and bundled llfix 1.0.7 remain unchanged.

## What the full upstream output says

The [website](https://www.llfix.net/) reports local encoding, serialization and
NIC enqueue latency, not RTT. The [pinned full output](https://github.com/CorewareLtd/fix-engine-benchmarks/blob/165cc999fa283a4ead17a7b935c5d3f850fec851/fix_client_benchmarks/llfix/results.txt)
also supplies a mean of 6,009.729945 ns and maximum of approximately 348.331 ms.
HP's three original Onload runs have means 1,937.624860, 1,992.899875 and
2,000.670489 ns, with maxima 49.5093, 58.9874 and 59.0739 ms. The vendor's better
p99 does not establish a better mean or extreme tail. Its raw output uses Onload
8.1.3.40 and a 3.2 GHz conversion; HP uses a different Onload generation and
Gold 6254/SFC9220 rather than Gold 6134/SFN8522 PLUS.

The original/framework p99 values agree within the observed run variation, and
the separate MONOTONIC_RAW framework series also has the same large tail. This
rules out the framework or a simple TSC conversion error as the sole explanation
of the approximately sevenfold p99 difference. It does not establish identical
clock calibration, hardware, compiler or vendor runtime inputs.

## Ordered samples and journal-page progression

Source checkpoint `4f4f8319ebf594f8c498df14792549ffe5d5dab3` adds a private,
read-only analyser. It verifies every consumed input against the original
854-file seal, requires the complete passing cold audit, aligns each measured
sample with its contiguous journal owner after 1,000 warmups, and checks the
published p50/p99/maximum against the complete ordered series. Four refusal and
alignment controls pass locally and on HP, with zero skips.

The analysis covers 12 million framework samples: six million per transport.
Logical first writes to a 4 KiB journal page occur in 6.2692% of calls in each
transport. This is a logical offset label, not an observation of a page fault.

| Calls above 10 microseconds | All | New journal page | Other | Other share |
| --- | ---: | ---: | ---: | ---: |
| Kernel | 2,363 | 1,543 | 820 | 34.70% |
| Onload | 64,561 | 5,253 | 59,308 | 91.86% |

The Onload non-page cohort still has p99 of 16.107–18.351 microseconds across
the six framework pairs. The page cohort has a worse tail, but logical page
progression cannot account for most of the calls above 10 microseconds.
Deliberately displaced page labels at offsets 1, 8 and 128 are retained as
controls. No offset is selected to improve the result. Per-second counts and
the largest sample indices/timestamps are retained for further diagnostics.

## Actual serialization control

Frozen controller source `555d9f136681bee2278c02a1885ec21a2a9c0125` imports only
the 39 hash/mode-verified helpers from the retained archive and verifies the
existing root payload and Onload runtime before execution. Four fresh pairs run
in balanced order: enabled, disabled, disabled, enabled. The only workload
change is the documented client `max_serialised_file_size=0` setting. The peer's
persistence remains enabled; owner, binary, CPUs, NICs and Onload defaults stay
the same. Live socket/options and process-fault monitoring make this a
diagnostic control, not an admitted timing comparison.

| Client serialization | p50 ns | p95 ns | p99 ns | Mean ns | Observed minor-fault delta |
| --- | ---: | ---: | ---: | ---: | ---: |
| Enabled, first | 646 | 2,237 | 17,817 | 1,927.851 | 88,502 |
| Disabled, first | 605 | 879 | 17,341 | 1,937.887 | 26,476 |
| Disabled, second | 605 | 876 | 18,198 | 1,936.163 | 28,009 |
| Enabled, second | 649 | 2,227 | 18,394 | 1,934.564 | 93,100 |

Median-of-two p50 improves 6.56% (647.5 to 605 ns); p95 improves 60.69%
(2,232 to 877.5 ns). P99 changes only 1.86% (18,105.5 to 17,769.5 ns), within
the observed pair variation. Serialization contributes ordinary-path overhead
and faults, but disabling it does not remove this p99 gap. Process sampling
includes startup/shutdown and does not assign a fault to a particular send call.

All four native pairs return zero and retain established accelerated physical
socket evidence and positive TX/RX deltas on both ports. The independent cold
audit passes all 4,004,000 received orders plus 2,002,000 client-persisted orders,
including actual required values, groups, checksums and contiguous owners. The
two disabled clients emit no message journals. Timing admission remains false.

## Onload configuration discrepancy and next controlled generation

All four live default stacks report `EF_SPIN_USEC=0`, `EF_BUZZ_USEC=0`,
`EF_MAX_PACKETS=32768`, `EF_MAX_RX_PACKETS=24576` and `EF_MAX_TX_PACKETS=24576`.
The vendor's [supplied runner profile](https://github.com/CorewareLtd/fix-engine-benchmarks/blob/165cc999fa283a4ead17a7b935c5d3f850fec851/common/onload_profile.cfg)
instead sets indefinite polling (`EF_POLL_USEC=-1`), 100,000 total packets,
75,000 receive packets and 25,000 transmit packets. This is a real input
discrepancy; the supplied profile is not an independent observation of the
vendor's live stack. Its causal contribution is still unmeasured.

Source `2e7e2de447ac39a578569e7870da071a56151049` prepares three new fixed
launchers: spin only, pool only, and both. Six focused controls pass on HP,
zero skips. The launchers preserve the original privilege drop, exact installed
payload and physical namespaces, keep both TCP loopback shortcuts disabled and
retain `EF_NO_FAIL=0`. The client-only eight-pair matrix uses all-enabled
serialization and balanced order: default/spin/pool/both/both/pool/spin/default.
The peer remains default to isolate the client's profile contribution.

The reviewed installer SHA256 is
`359133e64752f22175b643aa12146ef45972a3a60d5bb3f1bf380a257f83cd80`.
HP's home wrapper `~/install-llfix-gap.sh` verifies that digest before explicit
sudo installation. Current sudo rules permit running installed namespace
launchers but require the user's password to create these new protected files.
At this checkpoint installation and the profile matrix remain pending; no
profile result or vendor reproduction grade is inferred.

## Retention

The analyser binds original seal SHA256
`41479192ace42f1f2881d344dde59713d6b1d09c17891e86804b7b46e38d7ae1`.
New captures, cold audit, analysis, source capsules, fixed profile preparation
and controls are retained separately from the original archive. A first
source-capsule extraction guard refused the legitimate ancestor directory
members before extraction; a separate continuation checked safe paths/types
and all six bytes/modes successfully. That refusal remains recorded.

The new `hp-onload-gap-serialization1.tar.gz` archive is 83,303,513 bytes and
contains 83 sealed files. Independent streaming verification on HP and BEELINK2
checks every member's SHA256, size and mode and produces identical passing
receipts. Archive SHA256:
`0d771e2f112c6358f18afc4c739038b67a74669e555b14817e0f5a53cbf8804f`.
Seal SHA256:
`57601ea0b2a997371f268dd05b78ff2e84087aa4e705a8c104d353b1cd037a56`.

See the new [metadata selection](evidence/llfix-hp-onload-gap-20261006/selection-manifest.json)
for exact evidence paths and hashes. Raw journals remain in the sealed archive
under `~/archive/libhft/codex-llfix-reference-20261006/`; no old capture, SDK,
Stage 2 contract, prepared ranking or full-session generation is changed.