# C++ test verification and retained skip evidence

The C++ open-edition checker now retains every binary's stdout, stderr and exit
status instead of hiding successful output, including reported skips. It also
rejects a simulator that prints its expected banner and then crashes or hangs.
The corrected tool passes its 23 tests. The Linux open-edition sweep executes
146 C++ test binaries successfully, but two multicast round-trip subcases are
explicitly skipped. A separate sweep passes all 83 selected pro/venue binaries,
the HA-store probe, 107 venue-compiler tests and nine generated-header checks.
Both layering gates and the simulator checks pass. This is functional
verification, not a benchmark or an all-flavours conformance verdict.

## Reproduced checker defects

At `26cb08cb9`, with only the added checker regression tests, a successful
mock binary prints `SKIP: kernel resource unavailable`. The checker discards
it and prints a clean pass count. The new test fails because the skip is not
present in the output. The mock models the checker subprocess result, not a
real kernel-resource shortage.

A second test gives both simulator probes their expected startup banners but
sets their subprocess exits to -11, 1 or 2. The old tool reports PASS for all
three settings: it checks only for a banner, not whether the process exited
normally. All three subcases fail the new expectation. Source inspection also
shows that a timed-out simulator whose captured stdout contains the banner
can satisfy that same old predicate. The repaired test drives that timeout
case as well; no real simulator was made to crash or hang in this run.

Reviewing the retained real logs found a second diagnostic convention:
`test_fix_multicast` and `test_sbe_multicast_bus` print lowercase `[skip]` when
joined multicast traffic is not delivered. The initial detector counted only
uppercase markers, so its zero-skip summary was wrong. At `88e7f1ddc`, the added
lowercase-output tests fail, and three expected-refusal subcases (-11, 2 and 3)
also fail: the checker accepts the expected error text regardless of the
simulator's exit. These five assertion failures are archived separately.

## Checker changes

Each executed test binary gets a log containing its exit code and both output
streams. The default destination is the tool's dedicated out-of-tree build
root; `--test-output-dir` chooses a retained evidence directory. Failures and
successful or partially skipped runs are all kept. Uppercase SKIP/SKIPPED,
case-insensitive bracketed `[skip]`/`[skipped]`, and leading `skip:`/`skipped:`
marker lines are displayed, and the summary counts emitting binaries, not
skipped subcases. Narrative text about a message being skipped
does not count. The exit-zero pass count remains a process-exit count and is
documented as such; this does not detect an unreported skipped code branch.

The simulator must print the expected banner and exit 0, or exit 3 with its
explicit no-client-connected message. Exit 3 is the real simulator's normal
one-second no-peer outcome, not a generic success code; a child-socket setup
failure also returns 3 and must be rejected. Crash/error exits and the existing
60-second timeout fail regardless of captured banners. The timeout's stdout
and stderr are decoded and retained rather than discarded. Expected-refusal
probes use the same bounded timeout, must exit 1 with the refusal reason, and
also retain both streams and their exit status. A reason followed by a crash,
unexpected error code or timeout fails. No engine hot path or simulator
behaviour is changed.

## Verification results

All 23 checker tests pass. They cover the old verdict logic, visible successful
skips, retained stdout/stderr and nonzero exits, prefixed/stderr skip markers,
non-diagnostic narrative, banner-then-error exits, unexpected exit 3, and a
timeout after the expected banner, lowercase structured skip diagnostics,
unexpected refusal exits, refusal timeout and retained normal refusal output.
List mode still creates no output directory.
The successful unit and baseline failure logs are retained separately.

The full open-edition sweep builds and executes all 146 selected C++ test
binaries under stock GCC 14 C++11 flags. All exit 0. The retained logs identify
two partial skips: the actual multicast round trips in `test_fix_multicast`
and `test_sbe_multicast_bus`, while their other assertions execute. The core
gate passes with 51 CORE headers; the runner gate also passes. Both perform
actual loopback Logon round trips while testing their
include-tier boundary. The open simulator refuses the unavailable mmap store,
serves its metrics endpoint, and uses the file replay store; both no-peer
serve probes exit 3 with the expected reason. Their complete output is retained.

A bounded Python raw UDP control, independent of libhft, delivers its multicast
datagram on loopback on the first attempt, but receives none in 20 attempts on
the interface selected by the C++ tests. This supports a delivery-environment
gap; it does not identify the exact kernel, policy or driver cause or prove a
hardware transport healthy. The probes use an ephemeral UDP port and close
their own sockets, without changing routes, firewall rules or NIC configuration.
They run under a functional isolated-CPU lease overlapping the shared build;
neither is a latency measurement.

The separate default-development sweep builds and executes the Makefile's
26 pro and 57 venue binaries, all 83 with exit 0 and no explicit skip markers.
Its HA-store probe suite passes one test, the venue compiler suite passes 107,
and all nine checked-in generated headers match their descriptors. The driver
retains every output and bounds each binary to five minutes, each extra stage
to ten minutes, and the outer job to 45 minutes. These are functional checks:
the ef_vi test explicitly verifies header parsing with its live SDK path
compile-gated, and VMA ownership/poll tests use their fake API fixtures.
No physical bypass transport is exercised by this private sweep.

Together these sweeps execute 229 distinct C++ test binaries successfully; the
count is of processes, not unique assertions or all possible compiled branches.
They do not run every make correctness substage, the JNI/hftnet suites, NIC transports,
or a strict clock/latency campaign. Reference verification exits 0 and verifies
the fetched sources, captures and cached Maven dependencies; it also explicitly
reports the absent private brokerforge sibling as SKIPPED. The main checkout
is present elsewhere, but the checker looks beside this session's libhft
worktree. No checkout is moved, symlinked or edited to hide that distinction.
The cross-project fault-vocabulary check therefore remains outside this run.

The open and private sweeps use host-cpu-slot shared leases, queued behind
other sessions' exclusive jobs. Build products, fixtures and logs stay on /home,
and input hashes remain unchanged through each successful sweep. The separate
raw controls use an isolated lease. This was not an isolated latency measurement.
No installed binary, root staging helper,
machine configuration or benchmark result was changed.

## Evidence and source scope

`~/archive/libhft/cpp-open-evidence-20261004/baseline/` keeps the pre-fix unit
failures, patch and source hashes. `fixed/` retains the first complete successful
checker unit run. `full/` keeps the source pin and checker patch, tracked C++
and shared-fixture input manifest, reference check, build/gate output and all
146 per-binary logs plus the two simulator serve logs. Its tested tree is
`26cb08cb9` with the archived checker patch. `integrated/` repeats that sweep
at `c69fcd4f0`, after the C++ build-knob rename, and preserves the same two
lowercase skips despite its old zero-skip summary. `followup-baseline/` keeps
the five new assertion failures at `88e7f1ddc` plus the test patch;
`followup-fixed/` keeps the successful 23-test run. `multicast-controls/`
keeps the raw UDP source patch, hash and separate loopback/interface logs.

`final/` and `private/` qualify `88e7f1ddc` with the archived final checker patch,
after integration of the sparse-replay fixes. The final open sweep retains
146 binary logs and three simulator logs: normal refusal exit 1, and both
normal no-peer exits 3. It reports the two lowercase multicast skips correctly.
The private archive retains all 83 binary logs, its authoritative Makefile
selection, full build output and three additional-stage logs. Both input hash
checks pass. No C++ engine source is modified for this checker patch. The
portable open driver is
`internal/docs/bench/evidence/cpp-open-evidence-20261004/run-checks.sh`.
The private driver is `run-private-checks.sh` in that same directory; the raw
control is `multicast-control.py`, which takes a local IPv4 address and exits
0 only on delivery. Loopback exits 0; the selected-interface control exits 1.

The exact 229-binary checkpoint is retained on the pushed branch
`s/codex-libhft-hp-cpp-open-private-verified-20261004` at `0aa9df47a`, whose
tested engine/checker inputs match those manifests. The final publication
rebase also integrates `194d79e26`, the C++ provider lifecycle. Its changes
to the runner headers, tests and Makefile mean the older 229-binary result
must not be presented as a complete sweep of that newer tree.

`post-rebase/` independently qualifies the integration at `86dc3f7a6`:
23 checker tests, six runner/TLS suites in each of the development and open
editions, the additional pro reconnect suite, and both open layering gates
pass with unchanged input hashes. The six binaries are
`test_fix_reconnect_driver`, `test_fix_acceptor_connection_metrics`,
`test_fix_ssl_verify`, `test_fix_runner_realsocket`, `test_fix_acceptor_stop`
and `test_fix_proxy_initiator`; the thirteenth run is
`test_fix_reconnect_driver_pro`. None reports an explicit skip marker.
The selection still lists 146 open and 83 private binaries. This targeted
recheck is not a replacement all-binary sweep. Its portable driver is
`run-integration-checks.sh` beside the other two drivers.

The related Unix-listener fix and its cross-flavour findings are documented
in `RECORD_2026-10-04_hp-cpp-metrics-unix.md`. The wider review, managed
ownership fixes, non-Linux ownership verification and NIC campaign remain open.