# QuickFIX/n foreground production-session diagnostics — 2026-10-02

This implements a separate `--transport foreground` benchmark variant for
the pinned QuickFIX/n source `274b491d76f7755be85452dbefead08b2a6d62bd`.
It has canonical functional evidence, measured admin observations and raw
scheduler-delay samples. Physical Stage 5, common-fixture parity and the 10k/s
rate gate remain open.
The admitted DAC payload still uses the original threaded transport.

## Production session and declared transport

The adapter uses QuickFIX/n's ordinary configuration and SessionFactory,
public `Session.Next`, public `IResponder`, FIX44 message factory, production
stream parser and repeating-group dictionary. Session sequencing, logon,
heartbeat/TestRequest handling, logout and memory store are the production
implementation. No upstream source overlay is applied.

The public acceptor constructor creates the server session without starting
its socket reactor. An `AbstractInitiator` subclass uses `Start` to create the
client session normally; its temporary start thread returns immediately.
Socket polling, socket send and subsequent `Session.Next` calls run on the
foreground benchmark thread. This transport must be declared alongside any
future number. The default `--transport threaded` remains available.

Foreground mode requires two processes, one fixed internal run, a canonical
flow, numeric IPv4 endpoints, the FIX44 dictionary and disabled file logging. It rejects an adaptive
run or a same-process `both` invocation. The full controller is a local
diagnostic and cannot award Stage 5.

## Measurement changes

- Primary and scheduler-delay raw series are retained for every internal run,
  with canonical `quickfix_n-<flow>` identities, run indices and dropped counts.
  The CSV also contains every run's primary samples.
- Tick-to-trade starts after the production session constructs the complete
  FIX frame, performs its configured store work and encodes the socket bytes.
  The timestamp immediately precedes the first write of that prepared `W`.
  The owner is the server. The latest corrected build captures its end timestamp
  at production application callback entry, before dumping or field extraction;
  validation completes before the sample is accepted. Earlier builds captured
  the end after extraction, as the follow-up below records.
- `observations.json` records actual inbound admin callbacks and successful
  outbound admin socket sends during the measured window, including resend
  requests. Absent observation support is not filled with zero.
- The owner opens its window after the final pre-run warmup. The passive
  process opens after the final warmup response and closes after the final
  measured response. With zero warmup it opens at logon. The harness performs
  two warmup blocks for a fixed one-run pair; the effective count is recorded.
- `--admin-probe` sends a production TestRequest midway through the measured
  window and waits for the matching production Heartbeat. Its observations
  explicitly flag the injected probe, preventing report promotion.

## Retained checks

[Evidence](evidence/quickfix-n-foreground-20261002/) includes exact source
snapshots, build commands/logs, DLL/runtime hashes, canonical dumps, raw
series, admin observations and process output, sealed by `SHA256SUMS`.
Builds use SDK 10.0.112 with tiered compilation disabled. External reference
project outputs use `--artifacts-path` under the session build root; the
reference checkout stays read-only.

| Check | Flows | Result |
|---|---:|---|
| Framework-dependent foreground, warmup 5 / zero / admin probe | 12 pairs | all canonical, both processes exit zero |
| Self-contained foreground, warmup 5 / zero / admin probe | 12 pairs | all canonical, both processes exit zero |
| Default threaded transport, three internal runs | 4 flows | all canonical, 60 primary and 60 delay samples per flow |

Each foreground pair retains 20 primary and 20 scheduler-delay samples, with
no drops. Ordinary windows observe zero admin messages. Every injected probe
observes one inbound and one outbound admin message on both roles and zero
resend requests. Tick-to-trade records the prepared-wire clock declaration.
Both processes complete the production logon/logout exchange.

The first check used ten levels for tick-to-trade and failed its canonical
audit: that flow requires one bid/ask level. Its failed evidence is retained
under `initial-depth-mismatch`. The corrected adapter uses ten levels for MD
and mixed flow, and one level for tick-to-trade, as the existing contract
requires. These tiny WSL runs establish functionality; their JIT/startup
latency and achieved-rate readings are not performance claims.

## Reproduction

```sh
export LIBHFT_BUILD_ROOT="$HOME/build/codex-fullbench/libhft"
dotnet publish bench/dotnet/fixbench/quickfix-n/QuickFixN.Bench.csproj \
  -c Release -r linux-x64 --self-contained true -m:16 \
  --artifacts-path "$LIBHFT_BUILD_ROOT/quickfix-n-foreground-artifacts" \
  -o "$LIBHFT_BUILD_ROOT/quickfix-n-foreground-publish" \
  -p:QuickFixNSource="$HOME/code/quickfix-n" -p:UseSharedCompilation=false
python3 bench/cpp/fixbench/tools/open_source_fix_engines/check_quickfix_n_foreground.py \
  --app "$LIBHFT_BUILD_ROOT/quickfix-n-foreground-publish/QuickFixN.Bench" \
  --dictionary "$HOME/code/quickfix-n/spec/fix/FIX44.xml" \
  --output "$LIBHFT_BUILD_ROOT/quickfix-n-foreground-functional"
```

On HP or SCAN, inspect CPU-slot status and processes first and wrap the
outermost foreground build or controller with `host-cpu-slot wait`.
HP diagnostics use high priority. The medium local profile requires disjoint
process CPU masks and retains three fresh 10,000-sample pairs per flow after
1,000 requested warmups, matching the current .NET DAC report profile.
It applies the current raw/admin/rate/stability
checks and retains a mandatory local topology blocker. The latest controller
reports the actual quiet lease.

## HP rate diagnostics and polling correction

The initial 50,000-warmup attempt timed out in warmup and produced no measured
series. That failure and both process logs are retained under `hp/medium`.
The subsequent medium profile uses 1,000 requested warmups, as the .NET DAC
report controller does; it is not a controlled comparison to the first attempt.

Two adapter mistakes were corrected before the final local run:

1. Nonblocking socket sends/receives now use the error-code overload. A
   would-block result does not construct an exception in the polling loop.
2. The passive role polls continuously in foreground mode. Its inherited
   completion loop previously slept one millisecond after each poll; the
   original threaded transport's background I/O threads do not need this change.

Both corrected builds pass all twelve self-contained functional/admin checks.
The intermediate error-code-only medium run and the final continuous-poll run
each retain three fresh 10,000-sample pairs per flow. Every flow is canonical,
both processes exit zero, and the primary/admin/delay/tick observation checks
are complete. Both use a high-priority HP `isol` lease with disjoint masks
`8-12` / `13-17`, while another session runs shared conformance work. These
are localhost diagnostics with concurrent shared work, never Stage 5.

| Final continuous-poll flow | Achieved cycles/s, three runs | Mean p99, µs |
|---|---|---:|
| `nos_er` | 4083.847 / 4105.715 / 4106.601 | 251.431 |
| `md` | 2843.612 / 2836.159 / 2835.434 | 552.286 |
| `md_nos_er` | 1610.396 / 1609.551 / 1610.596 | 871.604 |
| `tick_to_trade` | 4008.181 / 3993.158 / 3985.863 | 239.927 |

All three percentile stability checks pass. Every flow fails the 98%-rate
gate and scheduler-delay interval gate. Its mandatory quiet-lease blocker is
also retained. No performance promotion follows from these diagnostics.

The controller now also seals a failure manifest when a process, raw series
or functional observation fails, returning nonzero without substituting
missing samples. A deliberately failing child process exercises that path;
its retained record is a controller check, not an engine measurement.

## Thread pinning and synchronous connect

The controller now supports explicit foreground CPUs within the disjoint
process masks, a requested warmup override and a declared subset of flows.
Every such run remains a local diagnostic. An initial pinned run, using
server CPU 12 and client CPU 16, improves some rates but still has rate and
stability failures. A separate 50,000-warmup NOS/tick experiment does not clear
those failures. Both sets are retained, including every raw series.

Foreground connect now uses the synchronous nonblocking socket API, a bounded
write-readiness poll and an explicit socket-error/connected check. It does
not create an asynchronous connect task. All twelve self-contained functional
checks still pass. The subsequent HP four-flow medium run uses CPUs 12/16,
1,000 requested warmups, three fresh pairs per flow and a high-priority
`isol-no-shared` lease. Its observations, dumps and 30,000 samples per flow
are retained under `hp/sync-connect`.

| Synchronous-connect flow | Achieved cycles/s, three runs | Local report checks |
|---|---|---|
| `nos_er` | 9945.149 / 9945.024 / 9943.500 | sample, raw/admin, rate, delay, stability pass |
| `md` | 5349.838 / 5374.579 / 5338.331 | rate and delay fail |
| `md_nos_er` | 3467.403 / 3475.708 / 3485.593 | rate and delay fail |
| `tick_to_trade` | 9939.776 / 9938.367 / 9939.735 | sample, raw/admin, rate, delay, stability pass |

Every flow passes the full percentile stability checks. The diagnostic
controller deliberately retains its Stage-5 lease blocker even under a quiet
local lease: localhost is not a physical DAC run. Shared work was present in
the earlier continuous-poll run and absent in this run, and thread placement
also changed. These are separate observations, not an isolated speedup claim.
The admitted physical payload is unchanged.

## Payload and callback-boundary follow-up

Source inspection after these runs identified two remaining adapter problems:
`nos_er` changes its order identifier but still sends constant side, quantity
and price, and the tick end clock is recorded after order extraction. The
current scenario contract requires varied order fields and callback-entry
`t1` before extraction. The table above reports only the listed local checks;
these builds cannot establish complete workload/timestamp parity or Stage 5.
Fresh evidence is required after those corrections.

A separate EventPipe MD diagnostic is retained under `hp/eventpipe-md`.
Its sampled thread time includes waiting threads and is not a CPU profile.
The textual method summary motivates inspecting snapshot group copying and
the adapter's duplicate book traversal; it does not establish their isolated
cost. The original binary trace is kept in the archive named by its metadata,
with its size and SHA-256. Profiling results are not performance comparisons.

## Checked payload and single book traversal

The corrected adapter varies `nos_er` side, quantity and price using its
existing declared tables. Snapshot extraction now uses QuickFIX/n's public
returning `GetGroup` overload, visits every entry once, folds its fields and
computes the touch used by the reactive order. No upstream source is changed.
Missing required order/report/book fields fail extraction; invalid depth,
entry type, size or crossed touch fails the row instead of folding zeroes.
The tick end clock is captured at the first instruction of `FromApp` and is
declared as `application_callback_entry_before_extraction`.

The standalone payload checker passes 31 checks with zero skips: a known
twenty-entry sink, touch selection, mutation of the last entry, absent required
fields and malformed books. All twelve self-contained functional/admin pairs
pass with 20 primary and 20 delay samples each. Each now retains twelve wire
cycles, individually audited, with varying ids across two digit widths,
alternating sides, varying prices/quantities, and current-touch price/quantity
derivation. All six MD role/warmup/probe sink comparisons match the previous
traversal. The existing two QuickFIX/n scenario regression checks pass.

These are functional diagnostics. The adapter tables differ from LibHFT Pure's
current generator tables; a common fixture and a cross-engine extraction-sink
comparison remain required before reporting a publishable comparison.
The fresh HP medium run completed at high priority under `isol-no-shared`.
Its immutable executable and controller snapshots are separate from ongoing
source edits. The controller now records the actual inherited quiet lease and
an independent mandatory `physical_topology` blocker for every localhost run.
The retained `hp/checked-book` result contains three fresh 10,000-sample pairs
per flow, after 1,000 requested warmups, on foreground CPUs 12/16 with process
masks `8-12` / `13-17`. Both processes exit zero in every pair. All primary
and delay samples, measured admin observations and percentile stability
checks are complete; its manifest records the quiet lease.

| Checked-book flow | Achieved cycles/s, three runs | Local report checks |
|---|---|---|
| `nos_er` | 9943.961 / 9940.877 / 9940.730 | sample, raw/admin, rate, delay, stability pass |
| `md` | 5588.500 / 5600.729 / 5592.355 | rate and delay fail |
| `md_nos_er` | 3695.852 / 3773.953 / 3776.716 | rate and delay fail |
| `tick_to_trade` | 9944.606 / 9946.111 / 9947.124 | sample, raw/admin, rate, delay, stability pass |

Every row retains `physical_topology`; MD and mixed flow also retain
`achieved_rate_below_98pct` and `scheduler_delay_above_interval`. The medium
run audits one cold cycle per pair; the separate functional checks above
audit twelve cycles and their changing fields. Common-fixture/sink alignment
remains open. This completed local diagnostic does not promote any Stage-5
or cross-engine comparison grade.

```sh
LIBHFT_BUILD_ROOT="$HOME/build/codex-fullbench/libhft" \
dotnet run --project bench/dotnet/fixbench/quickfix-n-payload-checks/QuickFixN.PayloadChecks.csproj \
  -c Release --artifacts-path "$HOME/build/codex-fullbench/libhft/quickfix-n-foreground-artifacts" \
  -p:QuickFixNSource="$HOME/code/quickfix-n" -p:UseSharedCompilation=false -m:16
```

## Foreground physical DAC integration

`run_scan_quickfix_n.py --foreground` selects a separate hash-pinned,
self-contained .NET 10 payload on either `hp-sfc` or `scan-mlx5`. Its fixed
namespace launcher pins foreground CPUs 12/16, bounds both root processes at
120 seconds, retains the correct raw sample owner and writes explicit measured
admin observations. Twelve cold wire cycles are individually audited for
variation and reactive derivation. The complete 203-file runtime tree and
seven named executable/dictionary/provenance inputs are checked before runs.

The `report` profile retains three independent 10,000-sample pairs per flow,
with physical-port TX deltas, prepared-wire start and callback-entry end clocks,
current raw/admin/rate/delay/stability gates and explicit transport metadata.
`comparison_fixture_unaligned` deliberately keeps it diagnostic while its
generator tables differ from the control. The older threaded payload remains
a separate variant. The foreground installer requires root admission on both
hosts; its `--check` mode verifies inputs without installing anything.
Those checks now pass on both hosts. The physical queue is held by each host's
`codex-fullbench-qfn-dac-20261002.service` user unit, waiting for its root install
marker before acquiring the quiet CPU lease. HP requests high priority; SCAN
requests normal priority. The matrix requires four complete physical tiny
checks, including both raw probes and explicit admin/tick observations, before
starting the full report profile. The separate local HP medium unit
`codex-fullbench-qfn-local-20261002.service` completed the checked-book run
above; its nonzero controller exit reports the remaining diagnostic blockers.
Queue source and admission-check
logs are retained in the evidence bundle.

Forty-eight focused DAC/payload/report regression checks pass, including
changed runtime hashes, invalid launcher inputs, missing/repeated wire cycles,
the existing report observation gates and comparison re-audit checks. A
foreground invocation without the selected host's quiet lease exits before
creating output or contacting that host.

## Outgoing mutable-book diagnostic

`--reuse-outgoing-book` is an opt-in foreground variant. It retains one
production `MarketDataSnapshotFullRefresh`, its groups and mutable fields.
Every send overwrites request id, symbol, all entry prices and all sizes from
the current request. QuickFIX/n's public field setters invalidate cached
field strings; its production session still constructs and sequences every
complete envelope. Serialized frames are not reused. The option rejects the
threaded transport, and observations declare `reused_mutable_fields` or
`fresh_objects`. The admitted DAC payload remains a separate build.

SCAN's SDK 10.0.111 built the separate self-contained payload with runtime
10.0.12. Eighty-five standalone payload checks pass with zero skips,
including full production serialization and complete extraction parity over
one- and ten-level books, changing identifiers/symbols and generator wraps.
All twelve functional/admin pairs pass, retaining twelve audited wire cycles
and both raw probes per pair. Their application fields and all twenty-four
role extraction sinks match the retained fresh-object functional baseline;
the comparison excludes envelope counters/checksums and time fields. Exact
inputs, build commands, output and parity checks are sealed under
`mutable-book` in the evidence bundle.

The same new binary completed two separate three-run medium arms on HP:
fresh objects first, mutable-book reuse second. Both use CPUs 12/16 under one
high-priority quiet lease, retaining 30,000 samples per flow per arm. This
keeps compiler/runtime changes out of the variant comparison. The persistent
user unit `codex-fullbench-qfn-reuse-book-20261002.service` acquired its quiet
lease after the existing shared conformance job finished. The complete
artifacts are retained under `hp/mutable-book`.

| HP flow | Fresh objects, cycles/s range | Reused book, cycles/s range |
|---|---:|---:|
| `nos_er` | 9940.297–9945.180 | 9939.565–9943.130 |
| `md` | 5527.973–5581.403 | 6140.908–6211.718 |
| `md_nos_er` | 3728.566–3764.117 | 4034.547–4047.682 |
| `tick_to_trade` | 9925.472–9944.034 | 9949.563–9950.702 |

Both arms have complete samples, canonical audits and admin observations,
and all percentile stability checks pass. NOS and tick-to-trade retain only
`physical_topology`. MD and mixed flow still fail the rate gate. The reused
MD arm passes its delay gate; fresh MD and both mixed arms fail it. These
sequential local arms do not establish an isolated causal speedup or a new
physical/comparison grade. Fixture alignment remains required.

SCAN completed the same two medium arms under one quiet lease, using masks
`10-13` / `14-17` and foreground CPUs 12/16. Each arm retains 30,000 primary
and 30,000 delay samples per flow, all canonical audits, complete admin
observations and passing percentile stability. Both use exactly the same
new binary. The arms ran sequentially, so these ranges do not establish an
isolated causal speedup. Their complete artifacts are under `scan/mutable-book`.

| SCAN flow | Fresh objects, cycles/s range | Reused book, cycles/s range |
|---|---:|---:|
| `nos_er` | 7134.088–7476.598 | 7076.336–7783.801 |
| `md` | 3529.539–3641.029 | 3857.732–3937.506 |
| `md_nos_er` | 2328.467–2405.325 | 2468.670–2542.190 |
| `tick_to_trade` | 6958.917–7242.218 | 7160.400–7486.689 |

Every flow in both arms fails the 98%-rate gate and retains
`physical_topology`. MD and mixed flow also fail the scheduler-delay interval
gate. Mutable-book reuse does not complete the report-rate requirement;
the complete HP result above also leaves MD and mixed flow below the rate gate.

```sh
ssh -t hp 'sudo /bin/bash /home/yann/code/libhft/private/ops/scripts/install-dac-quickfix-n-foreground-stage5.sh'
ssh -t scan 'sudo /bin/bash /home/yann/code/libhft/private/ops/scripts/install-dac-quickfix-n-foreground-stage5.sh'
# Inspect host-cpu-slot status and processes before this foreground run.
host-cpu-slot wait --pool isol-no-shared --priority high \
  --reason 'codex-fullbench: QuickFIX/n foreground physical DAC' -- \
  python3 bench/cpp/fixbench/tools/open_source_fix_engines/run_scan_quickfix_n.py \
  --rig hp-sfc --foreground --profile report --base-port 47150
```