fix(dynamic-codecs): honor variant.discriminator on the wire - #1128
fix(dynamic-codecs): honor variant.discriminator on the wire#1128plutohan wants to merge 2 commits into
Conversation
Enum codecs wrote the variant position regardless of the declared discriminator, so both directions disagreed with IDLs using custom discriminants. Each variant now encodes variant.discriminator ?? index via a constant prefix, and decoded objects carry __discriminator so the round trip is lossless. The argument validator and input transformer accept and strip the new field.
🦋 Changeset detectedLatest commit: 030379f The changes in this PR will be included in the next version bump. This PR includes changesets to release 5 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| const { __kind, ...rest } = input; | ||
| // The codec emits __discriminator on decode; drop it before re-encoding. | ||
| delete (rest as { __discriminator?: unknown }).__discriminator; |
There was a problem hiding this comment.
Thanks! nit: I'd rather we did something like:
const { __kind, __discriminator, ...rest } = input;I think you've already done it like that in the rest of the code. Btw, I'll do another thorough review soon after the AI review.
trevor-cortex
left a comment
There was a problem hiding this comment.
Implements the #1112 agreement: enum variant codecs become constant-prefixed payloads assembled with getUnionCodec, so the wire writes variant.discriminator ?? index, and decoded objects carry __discriminator for a lossless round trip. The input-path packages (dynamic-instructions validator, dynamic-address-resolution input transformer) strip the field before payload validation/re-encoding.
The core codec logic checks out: the union's byte-side lookup reads the size prefix and matches declared discriminants, the hidden-prefix constant then consumes/validates it, and the value-side lookup by __kind preserves prior encode semantics. Wire bytes are unchanged for default-discriminant IDLs, and the re-encode-decoded-output test locks in the round trip.
Things I'd like your take on (details inline):
- Mixed explicit/implicit discriminants —
variant.discriminator ?? indexuses the position for unannotated variants, but Rust continues from the last explicit value (enum { A = 5, B }→ B is 6, not 1). If the spec defines the default as the index and IDL producers (nodes-from-anchor etc.) always materialize explicit values in the mixed case, this is fine — worth confirming. - Error DX — unknown
__kindon encode and unknown wire discriminant on decode now surface asgetUnionCodec's generic out-of-range error instead of the discriminated-union codec's error that names the valid variants. - Changeset framing — the
**Breaking**label on a minor bump sits oddly next to CONTRIBUTING's "breaking changes to individual packages wait for the next major cut" policy.
One design note, not inline since it was agreed in #1112: encode silently ignores __discriminator even when it contradicts __kind (e.g. { __kind: 'Info', __discriminator: 20 } on a variant declared as 10 encodes 0x0a). That's the agreed contract, but a mismatch check would catch stale decoded objects being re-encoded after an IDL's discriminants change. Fine to punt.
For subsequent reviewers: worth a quick grep across the other dynamic packages for anything that consumes decoded enum shapes via exact equality or by iterating payload fields — only the validator and the input transformer were updated here, and the test fixture churn in dynamic-client shows how easily exact-shape assertions pick this up.
| // `{ __kind: 'Move', __discriminator: 1, x: 10, y: 20 }`. The wire | ||
| // byte honors `variant.discriminator`, falling back to the position. | ||
| const variants = node.variants ?? []; | ||
| const discriminators = variants.map((variant, index) => variant.discriminator ?? index); |
There was a problem hiding this comment.
variant.discriminator ?? index treats an unannotated variant's discriminant as its position. Rust (and Anchor) semantics continue from the last explicit value instead: enum Status { A = 5, B } gives B = 6, whereas this yields 1. If the Codama spec defines the default as the index and IDL producers always emit explicit discriminators once any variant deviates, this is correct as-is — but it would be good to confirm that's guaranteed, and perhaps add a mixed explicit/implicit case to the new test to pin the intended semantics either way.
| return getUnionCodec( | ||
| variantCodecs, | ||
| value => { | ||
| const kind = (value as { __kind?: unknown } | null)?.__kind; | ||
| return variants.findIndex(variant => pascalCase(variant.name) === kind); | ||
| }, | ||
| (bytes, offset) => { | ||
| const [discriminator] = size.read(bytes, offset); | ||
| return discriminators.indexOf(Number(discriminator)); | ||
| }, | ||
| ) as unknown as Codec<unknown>; |
There was a problem hiding this comment.
Minor DX regression vs getDiscriminatedUnionCodec: both lookup callbacks return -1 for unknown inputs, so an unrecognized __kind on encode or an undeclared wire discriminant on decode now throws the union codec's generic variant-out-of-range error, whereas the discriminated-union codec's error names the valid variants. Consider throwing a more descriptive error (listing valid kinds / declared discriminants) before returning -1.
Two smaller notes on the byte-side lookup: discriminators.indexOf(...) silently resolves duplicate declared discriminants to the first match — probably an invalid IDL upstream, but a cheap construction-time duplicate check would make the failure loud. And Number(discriminator) would lose precision for u64-sized discriminants above MAX_SAFE_INTEGER — vanishingly unlikely, just noting it's a known bound.
| // The codec emits __discriminator on decode; drop it before re-encoding. | ||
| delete (rest as { __discriminator?: unknown }).__discriminator; |
There was a problem hiding this comment.
Nit: this could fold into the existing destructuring on the line above instead of a cast + delete:
const { __kind, __discriminator: _discriminator, ...rest } = input as Record<string, unknown> & { __kind: unknown };(matching the pattern used in dynamic-instructions/src/validators.ts).
|
@trevor-cortex For your design note I think you're on to something. Since technically the variants are known at the time of creating the codec, are you saying we could bake the variants into that codec such that the actual byte value would be respected even when the decoded output is simple |
|
Yes — and the good news is the PR already does exactly that. The discriminator values are baked into each variant's codec at creation time (the constant prefix of My design note was about a narrower case: when the field is present and contradicts the baked-in value, e.g. On the rename: agreed that
No strong preference from me; if |
|
@trevor-cortex Oh I see in this case I think the current implementation is correct to ignore invalid values since the source of the truth is |
|
Agreed on ignoring mismatches — once On naming: your rationale is sound —
One thing worth a line in the README either way: the Codama node field is still Net: I'd go with |
|
@plutohan Ignore my conversation with Trevor haha. I'm happy to keep |
|
Done in 030379f, switched to the destructuring pattern matching validators.ts (with the same lint pragma, since the config flags the unused binding either way). On the two open notes from the review, for the record:
|
lorisleiva
left a comment
There was a problem hiding this comment.
Thanks! Don't worry about these follow up items. Would love @mikhd's approval on that one as well since it touches the dynamic packages.
Implements what we agreed in #1112: the wire honors
variant.discriminator, and decoded enums carry the discriminator so the round trip is lossless.variant.discriminator ?? index, assembled withgetUnionCodec(value lookup by__kind, byte lookup by reading the size prefix and matching declared discriminants). For IDLs without custom discriminants the wire bytes are unchanged.__discriminator(your naming), e.g.{ __kind: 'Info', __discriminator: 10 }. Encoding accepts and ignores the field, and the running example from the issue now round-trips:encode({ __kind: 'Critical' })writes0x1eanddecode(0x1e)returns{ __kind: 'Critical', __discriminator: 30 }.enumValueNoderesolution includes the field too, and the argument validator plus the codec input transformer strip it before payload validation and re-encoding, so decode-then-build keeps working.Tests: the issue's repro as a codec test (custom and gapped discriminants, struct variant payload, re-encoding decoded output byte-for-byte), updated decode expectations across the dynamic packages, and the client fixture assertions. All five dynamic package suites pass.
Changesets:
@codama/dynamic-codecsminor with a breaking note for the decoded shape, patches for the two input-path packages.Fixes #1112