# HP llfix paced stalls and allocation boundaries

2026-10-09. Read-only continuation of the
[physical-core placement campaign](RECORD_2026-10-09_llfix-hp-placement.md).
Eight retained paced pairs contain eight million primary samples and eight
million scheduler-delay samples. The original llfix 1.0.7 application, SDK,
Onload transport, installed payloads and measurements are unchanged.

## Findings

Two allocation boundaries are visible in the ordered data:

1. The original application's order pool starts with 2048 objects, then doubles
   when exhausted. It does not recycle those objects during the measured loop.
   Growth allocates/constructs a whole new block and grows its pointer vector,
   inside the primary timer. Of 59 primary calls at least 1 ms long, **37 occur
   exactly at these growth indices**. They contribute 274.207 ms of the total
   583.831 ms reported by those long primary calls: 46.97% of that duration.
2. The framework's primary and scheduler sample vectors also grow during the
   run. Their push operations fall outside the primary timer. All **23** delays
   at least 1 ms long that follow a primary call below 1 ms occur immediately
   after a sample-vector growth boundary. Their delay sum is 58.428 ms.

These are exact index associations supported by unmodified source and a compiled
functional growth probe. They do not assign every nanosecond to an allocator:
preemption, faults, transport and other work can occur during the same call.
The one-byte dummy orders in the functional probe establish growth indices,
not real Order construction cost. No measured sample is removed or reclassified
as an admitted engine latency.

The initial 2048-object description in the earlier profiles refers to starting
capacity. It must not be read as a fixed preallocated million-order working set.
The actual growth probe constructs 1048576 objects for 1000 warmup allocations
and one million measured allocations. This is the application's ObjectCache,
not llfix's separate internal outgoing-message cache.

**Next control:** preallocate the application order pool and framework sample
sinks in a separately named generation. Keep the original growing-pool result.
Further OS tuning alone cannot establish or remove these application/harness
allocation effects. The largest non-order-growth primary calls still range
from 16.942 to 55.848 ms; their cause remains unresolved.

## Sample alignment and wall accounting

The runner re-anchors its next deadline to the actual issue time. Therefore a
long primary call at index `i` generally affects scheduler delay at `i+1`, not
at `i`. At the 1 ms threshold, all 59 long primary calls have a long next delay;
the deliberately displaced same-index control finds only two overlaps.

For each run the final issue time relative to the loop origin is exactly
`N * interval + sum(scheduler_delay)`. At target 100000 calls/s the interval is
10000 ns. Across all eight runs the scheduler-delay sum is 1.243524112 seconds;
the remaining measured-wall accounting term is only 1671–1951 ns per run,
including final task/outer clock overhead. This verifies the ordering and the
rate-accounting relation against actual outputs.

Delays at least 1 ms account for 643.182 ms, or approximately 51.72% of the total
scheduler-delay sum. Rare events can materially affect achieved throughput while
being far below the 1% of samples that determines p99. This is a re-anchored
service-time workload: it does not reconstruct virtual arrivals lost during a
stall or become coordinated-omission-free.

Primary retains original RDTSCP/first-CPU-MHz conversion; the scheduler and wall
use CLOCK_MONOTONIC_RAW. No primary value is subtracted from a scheduler value,
and no frequency rescaling or calibrated vendor comparison is inferred.

## Every individual run

Counts use the fixed 1 ms threshold, not a fitted threshold. Other thresholds
and every exact event/index remain in the retained analysis.

| Client/server | Capture | Achieved calls/s | Scheduler sum ms | Primary calls >=1 ms | At order growth | Delays >=1 ms after vector growth | Largest non-order-growth primary ms |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 17/16 |02|98053.902|198.470476|7|5|3|55.848136|
| 16/17 |04|98652.159|136.623826|6|5|3|16.942000|
| 9/8 |06|98591.254|142.885816|8|5|3|18.086107|
| 8/9 |08|98470.480|155.325997|8|4|3|22.223034|
| 8/9 |09|98484.339|153.896716|7|4|3|23.816640|
| 9/8 |11|98473.024|155.063479|8|4|3|23.255787|
| 16/17 |13|98490.811|153.229761|7|5|2|21.620736|
| 17/16 |15|98541.296|148.028041|8|5|3|23.730345|

All three declared thresholds are retained:

| Threshold | Primary events | At order growth | Late-call events | After order growth | After vector growth | Other late-call events |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 100 us |96|64|145|64|48|33|
| 1 ms |59|37|82|37|23|22|
| 10 ms |22|9|22|9|0|13|

Other late-call events include calls following a long primary unrelated to
order-pool growth. Neither class is silently assigned to OS scheduling or TCP.

## Qualification and source identities

The first profile was declared before analysis at source
`025f47054b51342e6f0d5c5fb91f1850705382d7`. It validates 52 consumed files
against the existing placement seal, the complete passing cold audit's raw
hashes, and exact runner/platform/task sources. Full ordered CSV and binary
HFTBRW samples match the earlier audit. Initial and final boundary samples stay
explicit, and altered inputs/indices, malformed binary output and impossible
wall accounting are refused. Thirteen local and actual HP checks pass.

The allocation hypothesis was discovered from that first result. Its separate
[profile](../../bench/llfix_reference/diagnostics/PROFILE_PACED_GROWTH.md) and
source `511f1896cb570590915f4e9835ae2760dfb1783b` explicitly disclose that
sequence; it was not predeclared before the first analysis. The follow-up checks
54 placement files and two original upstream files against both existing seals.
A functional probe compiles against the actual immutable ObjectCache/allocator
and framework context. The selected GCC 11.5 executable exactly matches the
original placement preparation hash. Its receipt retains all 218 actual compile
input hashes, binary hash, compiler flags and observed capacity/construction
boundaries. Eighteen combined controls pass locally and on HP, zero skips.

Object growth is observed at measured indices 1048, 3096, 7192, 15384, 31768,
64536, 130072, 261144 and 523288. Sample-vector growth is separately observed at
powers of two; the primary retains capacity 1024 after discarded warmup. No
index tolerance or fitted displacement is used. The +/-1 displaced order labels
and all unexplained events are retained in the full follow-up result.

Both HP jobs use high-priority shared CPU leases with controller affinity 0–7.
The only native execution is the small non-timed allocation-index proof: no FIX
session, new timing run, root installer or OS configuration change is performed.
These diagnostics retain false timing/quiet/vendor/release admission and do
not alter the Stage 2 matrix or full-session qualification.

## Retained evidence

- [First ordered analysis](evidence/llfix-hp-paced-stalls-20261009/first/analysis.json).
- [Separate growth analysis](evidence/llfix-hp-paced-stalls-20261009/growth/growth-analysis.json).
- [Actual compiled probe](evidence/llfix-hp-paced-stalls-20261009/growth/probe.json).
- [Combined per-run summary](evidence/llfix-hp-paced-stalls-20261009/operations/summary.json).
- [Selection manifest](evidence/llfix-hp-paced-stalls-20261009/selection-manifest.json).

Both HP and BEELINK2 independently verify every member's content, size, mode and
membership in the new 53-file archive, retaining 603698 raw bytes and 130742
compressed bytes. Both keep
`~/archive/libhft/codex-llfix-reference-20261009/hp-paced-stalls1.tar.gz`.
Archive SHA256:
`993dc81df77ed76a577a7abc0078fcb0b55ddc622c70a25387d6e5d65ac000d1`.
Seal SHA256:
`7fc857cb13aa06891038f42c8c0e170e2191816f30d47d1cf2c890fb768bbea1`.

The first verification receipts performed the full byte checks but labelled
their result `sealed`, because the retention receipt overwrote the verifier's
status key. Those original receipts/script remain retained. A separate
one-expression continuation corrects only that label; the
[HP](evidence/llfix-hp-paced-stalls-20261009/hp-replica2.json) and
[BEELINK2](evidence/llfix-hp-paced-stalls-20261009/beelink-replica2.json) receipts
both say `replica-verified` and match all original archive checks. No source,
measurement, seal or archive byte changes.

The old 404-file placement archive and 854-file original source/runtime archive
remain preserved separately. This small continuation references their exact
seals rather than recopying 27 GB of unchanged journals/raw data.

After both replicas pass, [guarded cleanup](evidence/llfix-hp-paced-stalls-20261009/cleanup1.json)
removes only the two completed analysis materializations and their temporary
retention staging. It checks exact sealed file sets/hashes/modes, both replica
identities and process command lines before removal. The new primary archive,
old archives, installed payloads and active source/preparation caches remain.

The [integration verifier](evidence/llfix-hp-paced-stalls-20261009/verify-integration.py)
checks all seven qualified analysis/probe source files, the 21 selected sealed
metadata files, eight original supplements, the eight-run/count/accounting
invariants, 18 retained passing tests with zero skips, both corrected replica
receipts and every record link. The [receipt](evidence/llfix-hp-paced-stalls-20261009/integration1.json)
passes before publication. No fresh native benchmark is needed for integration.

After a normal fetch/rebase over concurrent C++ checkpoint and store regressions,
all seven source bytes/modes and all retained metadata match the qualified
generation. The [post-rebase receipt](evidence/llfix-hp-paced-stalls-20261009/integration2.json)
and 18 focused checks pass again with zero skips. Only this session's commits
are published; all concurrent changes are preserved.