# FAST codec correctness and allocation review — 2026-10-17

## Scope and fixes

Continuation of the deep review with three parallel reviewers and independent cross-review.
Session: `codex-review-codecs-1017`. Builds and tests ran serially under the host CPU lease,
with build parallelism capped at 16. Changes originate in the template compiler; checked-in
codecs and test fixtures are regenerated from their XML templates.

Nine confirmed defect groups are addressed:

- C++, Java and .NET byte-vector delta decoders preserve all eight bits of inserted bytes
  in both append and prepend forms. Only ASCII differences have their stop bits cleared.
- C++ byte/string delta prepend checks the incoming length without unsigned subtraction
  underflow. A difference larger than the 64-byte dictionary capacity is rejected before
  writing the temporary buffer.
- The generator rejects stateful byte/string initial values larger than that capacity.
  Regression cases cover mandatory and optional ASCII/byte-vector delta initials in the
  C++, Java and .NET emitters.
- Java and .NET byte/string delta encoders and decoders use the instruction's initial
  value when the dictionary slot is undefined. Tests independently encode the wire for
  `ABC → ABZ` and `RAW → RAX`.
- The C++ optional delta encoder leaves an undefined dictionary slot unchanged when
  encoding NULL. Initial-value seeding happens only for a non-null value.
- Java and .NET sequence-entry encoders invalidate the dictionary after a failed entry,
  including nested entries and failures after a written prefix. Further encoding is refused
  until reset, matching whole-message failure handling.
- C++, Java and .NET decimal exponent increment rejects `63 + 1 = 64` instead of wrapping
  to `-63`. The exponent must remain within `[-63, 63]` after applying its operator;
  increment wrapping follows the underlying integer type's limits. See
  [FAST 1.1 sections 6.2.2 and 6.3.6](https://www.fixtrading.org/wp-content/uploads/download-manager-files/FAST-Specification-1-x-1.pdf).
- Java and .NET optional numeric copy/increment decoding maintains the dictionary's null
  bit for explicit NULL and explicit non-null values. Omitted values retain NULL until an
  explicit value restores the assigned state.
- Java and .NET flat byte/string delta decoders eliminate per-call temporary allocation.
  Java reuses private scratch owned by the output object; .NET uses bounded stack storage.
  Java captures the subtraction value before reusing primitive scratch for a byte-vector
  difference.

Delta initial-value and subtraction/difference behavior follows
[FAST 1.1 sections 6.3.7, 6.3.7.3 and 6.3.7.5](https://www.fixtrading.org/wp-content/uploads/download-manager-files/FAST-Specification-1-x-1.pdf).
The previous local documentation describing exponent `63 → -63` wrapping is corrected.

Snapshot submitted to qualification: `e359057cbef6edccbefe93be905bcc074f404146`.
Parity qualification snapshot: `4fddd34ed8d33df35fe4455218c3ef6ffe5f5e35`. The engine tree is identical
to the codec qualification snapshot; the later change fixes diagnostic handling in the
parity harness. Final publication comparison is archived separately.

## Baseline reproduction

Regression sources were frozen at `74e88419d25d203d200cb0c616b352196c651160` with production
unchanged. Expected baseline outcomes were checked before applying the fixes:

| Selection | Baseline result |
| --- | --- |
| Java new regressions and native guard | 27 failed; 10 passed; zero errors/skips |
| .NET new regressions and native guard | 27 failed; 10 passed; zero errors/skips |
| C++ delta boundary fixture | 6 failed checks; 70 passed checks |
| C++ decimal increment fixture | 8 failed; 2 passed |
| C++ oversized prepend under AddressSanitizer | 4 stack-buffer-overflow reproductions |
| Generator unit suite | 12 expected failing subcases within 150 test methods |
| Rust decimal increment integration controls | 10 passed; zero failures/ignored |

Each managed baseline comprises 16 byte-delta cases (9 failed), 10 sequence-entry cases
(9 failed), 10 decimal increment cases (9 failed), and one passing required-native guard.
The Rust controls already implement the expected exponent behavior; this batch changes no
Rust production codec logic.

Tests use independent primitive-level wire construction, compare exact bytes, exercise
append/prepend and the 64/65-byte boundary, and verify dictionary validity and reset/reuse.
Sequence cases cover zero-progress and partial-prefix failures in outer and nested entries.
Decimal controls cover mandatory/optional instructions, undefined initial state, `62 → 63`,
and NULL propagation. The fixed controls additionally check recovery from NULL through
explicit `62` and omitted `63`.

Allocation measurements reuse the wire, dictionary and output objects after 20,000 warm-up
decodes. Each measured loop decodes 100,000 messages containing four non-null delta fields:

| Runtime | Baseline allocated bytes | Bytes per message | Fixed result |
| --- | ---: | ---: | --- |
| Java | 41,600,000 | 416 | 0 bytes, both editions |
| .NET | 35,200,000 | 352 | 0 bytes, both editions |

Java uses thread allocation accounting and .NET uses `GC.GetAllocatedBytesForCurrentThread`.
The fixed assertions require zero allocation in this reusable flat decoder. These are
allocation measurements; no end-to-end latency improvement is claimed.

## Fixed qualification

| Selection | Development edition | Open edition |
| --- | ---: | ---: |
| C++ FAST correctness programs | 19 passed | 19 passed |
| Java FAST selection and native guard | 51 passed | 51 passed |
| .NET FAST selection and native guard | 95 passed | 95 passed |
| Rust FAST library tests | 75 passed | 75 passed |
| Rust FAST integration tests | 25 passed | 25 passed |

All selections had zero failures, errors, skips or ignored cases. All four managed
required-native guards passed. Open .NET loaded the canonical `native-open` output
through `LIBHFTNET_LIBRARY`. Rust integration selection includes the 10 new exponent
controls, 12 runtime-failure cases and 3 presence-map descriptor cases.

In each edition, the C++ corpus programs decoded and re-encoded all 5,020 goFAST
and 30,001 mFAST messages byte for byte. The generator suite passed all 150 methods,
and generated-source drift checks passed. The new C++ delta fixture passed 76 checks
and the decimal fixture passed 10 cases under AddressSanitizer and fatal
UndefinedBehaviorSanitizer. All four oversized-prepend sanitizer controls passed.

The first cross-surface parity run exposed a harness defect: successful process stderr
was appended to stdout before comparison, so the JVM's `JAVA_TOOL_OPTIONS` notice
made 19 matching Java output lines appear different. The helper now forwards diagnostic
stderr separately and compares successful stdout unchanged; failed processes retain
both channels for error reporting. Its three subprocess tests reproduce two baseline
failures and pass after the fix, including preservation of `ENCODE-FAILED` on stdout.

Cross-surface parity passed all 19 output lines across C++, Java, .NET and Rust, with
the existing wire fingerprint unchanged. Native-flavour parity passed for Java/JNI and
.NET Native against their managed counterparts. Its scope includes primitive boundaries,
registry addressability, chained/nested sequences and the three-message CQG capture;
the captured tool log records the precise template coverage. Both parity-harness unit
suites passed (3 cross-surface cases and 4 native-diagnostic cases). The changed-engine
export scan and whitespace checks passed.

After runtime qualification, trailing whitespace was removed from six generated Java
files and the emitter line that produced it. Normalized source comparisons prove those
output changes are formatting only. Generator tests and drift checks passed again.
The final rebase carries unrelated FIX/session changes and shared-fixture relocation;
all 165 FAST source/input files retain the qualified content after that normalization.
The final source proof records the build-metadata review and exact file hashes.

This is targeted FAST qualification, including existing mFAST/goFAST corpus programs.
Windows toolchains, live production feeds and a full repository test run are outside this
batch's qualification scope.

## Publication and evidence

The final source comparison and publication hashes are recorded in the archive.
Archive: `~/archive/libhft/reviews/2026-10-17-codex-review-codecs-1017/`.
Keep raw logs, XML/TRX reports, commands, source bundles, patch checksums and review notes
outside Git. Key records are `baseline-summary.json`, `qualification-summary.json`,
`stages.jsonl`, `patches-approved.json`, `rebase-proof.json`, and the final source/publication
proofs.

A shared dictionary key used by optional and mandatory instructions may retain stale
empty-state metadata when a mandatory explicit copy overwrites a prior optional NULL.
That observation has no runtime reproducer in this batch and remains a separate follow-up;
it is distinct from the confirmed managed optional numeric copy/increment defect above.