# HP llfix: receiver journal rotation investigation

2026-10-09. Follow-up to the
[preallocation results](RECORD_2026-10-09_llfix-hp-preallocation-results.md).
That campaign removed the known allocation spikes but retained periodic long
sends. This generation investigates receiver persistence and tests an explicit
receiver journal size with the unchanged installed binaries.

## Original source and hypothesis

The original SDK1.0.7 uses a default64MiB journal. The original sender config
sets640MiB, while the receiver retains the64MiB default. The receiver therefore
creates four incoming files in each retained million-call pair. Source and
file inventory inspection motivated the exploratory
[analysis profile](../../bench/llfix_reference/diagnostics/journal_rollover_analysis/PROFILE.md).

`MessageSerialiser::write` checks the remaining file bytes before recording a
frame timestamp. Its synchronous `switch_to_next_memory_mapped_file` flushes
the old mapping, closes it, then opens the next file. On Linux,
`MemoryMappedFile::flush_to_disc` uses `msync(..., MS_SYNC)`. Creating a missing
file allocates a file-size buffer, zero-fills it, writes all bytes, frees it,
then maps the file. These operations run on the receiver's processing path.

The journals stamp `clock_gettime(CLOCK_REALTIME)`. Adjacent sender journal
timestamps bracket post-send persistence records; they are not exact primary
timer entry/exit timestamps. Primary RDTSCP/first CPU MHz and scheduler
MONOTONIC_RAW remain separate, unscaled clock domains. No duration is
subtracted across those domains.

## Retained analysis

Source5d31a3d858d07890adb359cd94efa486d21083ce read all eight pairs but refused
the inline `seal.json` member after processing the data. The original failed
log and controller receipt remain unchanged. Source49735722b adds an exact
inline/external seal-byte, size and mode check, with altered-input controls.
The old archive/source/runtime is unchanged.

The corrected reader passes all eight runs,89 selected sealed inputs and five
exact SDK headers, including their actual dependency identities in all four
compiled sender variants. It parses16016000 bilateral order records, checks
contiguous owners across every file and identical sender/receiver order-wire
digests. Archive and inline/external seal identities pass before/after reading.
Eight focused reader checks and seven sizing-controller checks pass locally
and on HP, zero skips. Frozen-source/installed-input checks also pass.

All24 receiver transitions exhaust the old file's remaining bytes:3 bytes at
the first boundary and245 at each later boundary, less than the next record.
The first owners in files2/3/4 are261989,523112,784235 in every run. Their
last-old/first-new timestamp gaps range31.200220–68.691082ms. Every transition
overlaps at least one >=1ms send's adjacent journal timestamp interval.

There are43 long calls across the eight runs.32 overlap a receiver transition;
the11 others are at previously proven application order-growth indices. Some
growth calls also overlap a transition, so overlap alone does not assign their
duration to receiver rotation. All13 long calls in orders/both, where order
growth is disabled, overlap a transition. In the two both-preallocated runs,
all six remaining long calls overlap the three receiver rotations:

| Run | First owner in new receiver file | Receiver epoch gap ms | Sender measured index | Primary original ms |
| --- | ---: | ---: | ---: | ---: |
|04-both|261989|68.691082|264195|36.475738|
|04-both|523112|48.645477|525318|16.495125|
|04-both|784235|55.108768|786424|22.947881|
|05-both|261989|55.222134|264190|23.106349|
|05-both|523112|50.486469|525319|18.329361|
|05-both|784235|55.003592|786424|22.864708|

Those sender owners are3190–3208 ahead of the first owner in the new receiver
file. Buffering lets sends continue during a receiver pause before backpressure
can reach the sender; this explains why the long-send index follows the file
boundary rather than matching it. This is an inference from source, ordering
and timestamps, without an event-aligned socket/syscall trace. The evidence
does not partition flushing, file creation, copying, page faults, preemption or
transport waiting per call. The two table durations are distinct clocks.

## Receiver sizing control

The [declared profile](../../bench/llfix_reference/diagnostics/journal_rollover_controls/PROFILE.md)
at sourcec22212e752416f5ec85557b16ad975e7cb52f6a2 fixes six fresh pairs in order
64,640,640,64,64,640MiB. The original installed both-preallocated sender and
receiver are reused. No build or root installation is required. Only the
receiver SESSION1 size property changes. All other configuration bytes and
the whole capture function AST reverse exactly to the old source.

Every pair uses1000000 measured calls,1000 warmups and100000calls/s, native
client/server CPU17/16, observer/controller CPU0-7, receiver-only Onload spin,
default receive buffer and physical SFC9220 DAC namespaces. Installed
generation28f4ccd4a65f, original SDK1.0.7, GCC11.5, primary timed-loop bytes,
wire fields, record format, persistence and OS policies remain pinned.

Larger files are initialized before measurement. This also changes startup
writes and cache/dirty-memory conditions, so it does not isolate every aspect
of rotation cost or establish new persistence guarantees. The complete
original cold audit is reused, with additional actual file-topology and exact
config checks. Original failures are retained separately from continuations.

**A larger file postpones rotation; it does not make rotation inexpensive.**
An ongoing640MiB session eventually reaches its next boundary, with a larger
file to flush and create. These fixed million-call runs test the presence or
absence of rotation during this window, and cannot establish long-duration
tail behavior. Preparing the next file ahead of time or moving appropriate
persistence work off the receive path would be separate SDK changes, requiring
their own durability and performance qualification.

## Complete matched results

All six physical pairs and complete independent cold audits pass:6000000
primary samples,6000000 scheduler samples and12012000 persisted bilateral
endpoint order records. Required values, groups, owners, envelopes, raw CSV/
HFTBRW, clocks/counts, actual affinity, Onload sockets and bilateral physical
counters pass; no skips or nonzero native return codes. Receiver buffer is
131400 bytes in every run. The64MiB receiver has four incoming files; the640MiB
receiver has one. Persistence remains enabled in both arms.

The statistics below are medians of three individual run statistics, with the
largest true maximum across those runs. Quantiles are not pooled and no
confidence interval or general performance winner is inferred.

| Receiver file size | p50 us | p95 us | p99 us | Mean us | Worst ms | Achieved calls/s | Calls >=1ms across3M |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|64MiB|1.457|2.932|3.672|1.621985|22.585136|98821.15|9|
|640MiB|1.440|2.937|3.711|1.555580|0.123939|99307.70|0|

The nine64MiB long calls again occur near264.2k,525.3k,786.4k: three per run.
No640MiB call reaches1ms. Worst latency falls by about182x in this fixed
window. The rarity of these events (9/3000000=0.0003%) explains why eliminating
them does not improve p99: the larger-file point estimate is3.711us versus
3.672us. The three-run scheduler-delay sums are355.438181ms and214.885538ms,
respectively. Their exact wall-accounting remainders are3480–4991ns, including
the previously disclosed bounded capacity/count observations.

| Condition | p99 us | Actual maximum ms | Calls >=1ms | Scheduler sum ms | Achieved calls/s | First CPU frequency Hz |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|01-64mib|3.659|16.356150|3|105.268645|98958.239|3101000000|
|02-640mib|4.027|0.123939|0|76.895453|99236.868|3099000000|
|03-640mib|3.669|0.065938|0|69.707271|99307.704|3102000000|
|04-64mib|3.672|20.742024|3|119.286944|98821.148|3105000000|
|05-64mib|3.673|22.585136|3|130.882592|98708.037|3100000000|
|06-640mib|3.711|0.035908|0|68.282814|99321.768|3100000000|

The original timestamp associations, synchronous SDK rotation path and matched
size control strongly support receiver journal rotation/backpressure as the
source of the periodic millisecond sender tail. They do not partition the
individual flush/create/preemption/socket costs or explain the separate p99
gap against vendor results. No source implementation is changed or fixed.

The [ordered raw analysis](evidence/llfix-hp-journal-rollover-20261009/continuation/sizing-analysis.json)
independently rechecks all CSV/binary/count/statistics and every long event,
using the exact prior helper, without dropping events or clock rescaling.
The [original eight-pair association](evidence/llfix-hp-journal-rollover-20261009/continuation/analysis.json)
and [six-pair cold audit](evidence/llfix-hp-journal-rollover-20261009/capture/journal-sizing-audit.json)
retain the complete details. Source/support bytes and modes remain identical
after the source publication rebase at8877d7fe4, preserving the concurrent Rust
wire repair. The original source pins and installed generation remain explicit.

## Preservation

Complete original failure, corrected analysis/source, unchanged-binary capture,
all raw samples/journals, actual configuration and full cold receipts are
sealed together. Both HP and BEELINK2 independently verify every member's
hash, size, mode and membership:163 files,13277305978 raw bytes,281637658
compressed bytes. The archive is retained on both hosts under
`~/archive/libhft/codex-llfix-reference-20261009/hp-journal-sizing1/`.

- Archive SHA256:`d0165c6fd38c0610d1a1471685764111775c232c38f3ee60ee08c9a76bf30b2a`.
- Seal SHA256:`af22d074585e064cd0c14c34d38fb18b4c2229b9a66819e96a2c5c314d04b515`.
- [HP verification](evidence/llfix-hp-journal-rollover-20261009/hp-replica.json).
- [BEELINK2 verification](evidence/llfix-hp-journal-rollover-20261009/beelink-replica.json).
- [Original failed reader](evidence/llfix-hp-journal-rollover-20261009/failed-analysis/analysis.log).
- [Original failed controller](evidence/llfix-hp-journal-rollover-20261009/failed-analysis/run.json).
- [Corrected controller](evidence/llfix-hp-journal-rollover-20261009/continuation/run.json).
- [Source preservation](evidence/llfix-hp-journal-rollover-20261009/continuation/source-integration1.json).

Raw CSV/HFTBRW and padded journals remain in the sealed archives. Source,
metadata, original failures and report inputs are selected into the private
evidence directory. The [integration verifier](evidence/llfix-hp-journal-rollover-20261009/verify-integration.py)
checks selected sealed bytes/modes, every report table and all eight qualified
source/support files. Historical generations, installed payloads and active
preparation/functional/compiler inputs are preserved.

After both archive verifications,
[guarded cleanup](evidence/llfix-hp-journal-rollover-20261009/cleanup.json)
checks all163 completed materialized files (10 failed-analysis,34 continuation,
119 capture), preserves the detached retention log, refuses active consumers,
then removes only those three own temporary roots. No installed payload,
SDK/cache, original native/functional input or other session's job is changed.

The original eight-pair campaign and this sizing generation remain observed
diagnostics. Quiet timing, vendor reproduction, Stage2, full-session and
release admission remain false. The two native primary-clock and scheduler
domains are preserved, including the pacing's coordinated-omission limitation.