# Cross-benchmark measurement lessons and SCAN IRQ review

The [HP journal experiment](RECORD_2026-10-09_llfix-hp-journal-rollover.md)
showed three rotation-related millisecond events per million sends that p99
could hide. The existing CDF budget/compression and C++ cold sample reservation
are already published in `1091b38fa` and `f76eb4fb5`. This continuation implements
portable ordered-tail, measurement-receipt and machine-profile checks plus the
latency notebook integration. It does not copy HP tuning into SCAN.

## Implemented checks

`bench/common/measurement_audit.py` reads ordered HFTBRW1 u32/u64 and raw CSV,
including gzip inputs. It retains exact integer maxima and ordinals, inclusive
100-us / 1-ms / 10-ms counts, and sixteen largest observations per file/run/probe.
Counts use the complete input, independently of the CDF plotting budget.
It refuses malformed formats, truncated/trailing binary data, missing/reordered
CSV indices and fractional nanoseconds; decimal unit conversion does not round
away a small fractional tail. A sixteen-million-sample per-series resource limit
is explicit and never a benchmark capacity verdict.

Dropped samples, empty series and a possibly clamped u32 ceiling are visible.
A CSV exported from a u32 buffer cannot hide its ceiling. The utility does not
restore observations already lost by an older writer. Missing expected counts
remain a failure when the declared measurement contract is checked.

The contract checks both endpoint store/logging/buffer/journal/allocation/
first-touch/CPU receipts, every expected series/count and pacing mode. Paced
profiles require owner elapsed time, offered rate, completed/planned/discarded/
pending arrivals and retained scheduler delay. Achieved rate is computed from
the supplied owner elapsed time and completed count. Old reanchored scheduling
is unchanged; absent counters cannot be invented from service latency.

Boundary/endurance profiles need actual supplied measured event ordinals for
specified endpoint lifecycle events. Duplicate event copies cannot inflate
coverage. These are checks of supplied receipts, not independent qualification
of an engine's instrumentation or clock correlation. All outputs preserve
`benchmark_admitted=false` and `coordinated_omission_correction_verified=false`.
The new synthetic public examples demonstrate the completeness gate.

The latency notebook automatically writes `data/tail-diagnostics.json` and
renders a separate table of retained maxima/ordinals and stall counts. It keeps
integer values even with an empty series beside a value above 2^53. Summary-only
and HDR-only evidence cannot recover ordered counts. Existing summary statistics,
mean-run maxima, pooled distributions, numerical selectors/rankings, historical
prose, Stage 2 three-probe calculations and renderer source are preserved.
The binary reader now rejects unknown value formats rather than assuming u64.

## Software qualification

The complete stripped selection materializes **768 files** (765 selected sources
plus three templates), without `.git`, an engine or private context. HP used the
existing read-only Python 3.9.25 notebook environment under a **high-priority
shared CPU lease**. The final generation passes **42 tests, zero skips**, including
all **58 actual notebook code cells** and the full-width/empty-series display
control. All 768 source bytes/modes remain unchanged after execution. Earlier
passing generations and all their input maps are retained separately.

Local retained compatibility checks pass two tests: original CSV contents and
Markdown replay exactly, and 57 of 61 existing generator function ASTs remain
unchanged against `f76eb4fb5`. Four functions change for strict decoding, the
sidecar write, supplemental display and CLI integration; one collection function
is added. The renderer is byte-identical. The final raw utility has seventeen
focused checks, including subnanosecond/huge decimal, u64, drop, ceiling,
ordinal, boundary, pacing and fresh-output refusal controls.

A functional C++ program uses the **actual unchanged HftBenchRawSamples.h writer**
with GCC 15.2.0 / O3 / C++11. A 100,000-sample vector contains exactly one
22,585,136-ns observation at ordinal 73,429. The new audit recovers all three
threshold counts as one and retains that ordinal. Appending UINT64_MAX exercises
the original writer's u32 clamp; the audit marks its maximum incomplete. The old
and new report readers produce identical legacy fields/float values for **200,001
observations** across both files. This is a writer/decoder proof, with no engine,
network session or performance result.

[Verification](evidence/measurement-lessons-20261010/verification.json) binds 27
source/support inputs. [The verifier](evidence/measurement-lessons-20261010/verify.py)
checks source bytes/modes, original-function scope, final HP receipt and the
ordered writer controls. The complete archive mappings, original generator and
original/intermediate/final logs remain in the evidence directory. The three
complete source maps are stored as lossless `.json.gz` files; no provenance
entries are removed for the size reduction. Public export
readiness has zero scrub/private/unresolved/dead-link findings and 66 existing
textual escapes. This does not add standalone SDK or release admission.

## SCAN review and correction prepared

Read-only live review confirms isolated CPUs 10–19, housekeeping 0–9, twenty
unique physical cores with no observed SMT sharing, performance governors on the
single server/client CPUs, and busy_read/busy_poll=50/50. The installed load
profile is byte-identical to the source profile. Application/server/client/VMA/
pool-coordinator sets are disjoint as declared; those policies stay unchanged.

The deeper IRQ readback finds **twenty Mellanox completion IRQs** with effective
CPU placement on isolated CPUs 10–19: IRQs 99–108 and 118–127. This includes CPU10
(single server) and CPU15 (single client). Historical `/proc/interrupts` counters
are not a measurement-window delta and do not quantify a performance effect.
The kernel command line's `irqaffinity=0-9` does not establish the current
per-IRQ placement. irqbalance is inactive.

`bench/load/machine_profile_audit.py` checks live isolation, online CPUs, SMT,
endpoint/coordinator/maintenance placement, busy-poll values and a selected NIC
IRQ pattern's effective affinity. All twenty wrong IRQs fail this read-only
preflight. The companion IRQ setter plans only the incorrect IRQs; already
correct IRQs stay untouched. It verifies changed IRQ identities/requested masks
and effective readbacks and restores this invocation's original requested masks
if a write/readback fails. Five fixture tests include stale identity, stale
mask and failed-readback rollback controls; no fixture writes a real host.

The new private `internal/bench/load/scan-nic-irq-policy.json` selects Mellanox
IRQs and housekeeping CPUs. The existing load profile/loader and frozen campaign
pins stay unchanged. The reviewed source/profile capsule is staged at:

`/home/yann/.cache/libhft/scan-benchmark-irq-policy-v1-12b03cda96fc`

Seal: `12b03cda96fcba90c558c3aa130b461c045602d6791d6f87103e5342ca6ab7b9`.

`/home/yann/apply-libhft-benchmark-irqs.sh` checks that seal and the installed
profile, asks for sudo authentication, then waits for a **high-priority
isol-no-shared lease** before applying/retaining before/apply/after receipts.
It refuses an active irqbalance daemon. No other owner's job is stopped.
Non-interactive sudo refused because a password is required, so the live IRQ
change is **pending the user running that script**; the prepared plan is not
an applied-policy receipt. The setting is intentionally nonpersistent and must
be rechecked after reboot, driver reload or queue reconfiguration. Starting a
new timing epoch and repeating matched floors/functional/placement/pacing checks
is required before comparing new measurements with earlier SCAN results.

No SDK, canonical wire, shared clock/build/layout API, Stage 2 adapter/recipe,
installed runtime, old source capsule, raw archive or ranking is modified.

## Publication and retention

Source and functional evidence are on private main `b5f2b9e89`, after a normal
fetch/rebase preserving the concurrent JNI acceptor change. The qualified and
integrated source refs retain `e8b59fae6` and `b5f2b9e89` respectively. All 768
qualified exported bytes/modes match the integrated tree, and all 27 focused
inputs pass the verifier. The first extra map-comparison wrapper omitted the
release helper import path; its inert loader was corrected and the comparison
passed without changing harness source or any grade.

Both HP functional generations are sealed under
`~/archive/libhft/codex-llfix-reference-20261010/measurement-lessons1/`.
Each archive verifies all 768 source bytes/modes before the two completed HP
build/source capsules are removed. The retention/cleanup receipts are retained
with the record. SCAN has no IRQ application receipt at publication: its reviewed
cache capsule/home wrapper remains available for the pending sudo step.