ML-KEM: ASN.1 Module - add BOTH private key format and validate expanded keys

[jakemas]

Important

Stacked on #2709 — please merge that first. This branch is #2709's branch plus three commits: ML-KEM: parse the RFC 9935 both private key format, ML-KEM: validate expanded keys when parsing a private key, and ML-KEM: reject trailing data after the private key CHOICE. The second needs KEM_check_key and the EVP_R_* reason codes that #2709 adds, so it does not build without it.

#2709 is fork-based, so GitHub will not let me set it as this PR's base branch; the diff below therefore currently includes #2709's changes as well. Once #2709 merges to main the diff here collapses to just the three commits above on its own. To review only this change in the meantime: git diff <#2709 head>..<this head>, or read the three commits individually.

Related issues

Related to #2416 — the ML-DSA counterpart of the both parsing added here.
Related to #2709 — this PR consumes the KEM_check_key machinery it adds.

Context and motivation

RFC 9935 section 6 defines three ML-KEM-XX-PrivateKey CHOICE encodings:

  ML-KEM-512-PrivateKey ::= CHOICE {
    seed [0] OCTET STRING (SIZE (64)),
    expandedKey OCTET STRING (SIZE (1632)),
    both SEQUENCE {
      seed OCTET STRING (SIZE (64)),
      expandedKey OCTET STRING (SIZE (1632))
      }
    }

Two gaps, which together are why the RFC's four "bad private key" examples in Appendix C.4.1 all imported cleanly before this change:

1. both was not implemented. It was stubbed with a TODO in kem_priv_decode and rejected outright. The format exists so a producer can serve peers that support only one of the two representations, so we should be able to consume it.
2. Expanded keys were accepted verbatim. Section 8 defers to FIPS 203 section 7.3, which requires a "hash check" before an expanded decapsulation key is used. We did not perform one at import, so a corrupted expandedKey was accepted and failed — or silently misbehaved — only later.

Description of changes

both CHOICE. Added to kem_priv_decode, dispatched on the SEQUENCE (0x30) tag alongside the existing [0] (0x80) and OCTET STRING (0x04) cases, with seed and expandedKey lengths validated against the parameter set. KEM_KEY_set_raw_keypair_from_both then performs the seed consistency check required by section 8: the expanded key is regenerated from the seed via ML-KEM.KeyGen_internal(d, z) and compared bytewise against the presented expandedKey.

The bytewise comparison is load-bearing, and this differs from the ML-DSA both handling in #2416, which compares derived public keys. For ML-KEM a public-key comparison is not enough: an expandedKey that differs from the seed only in z, the implicit rejection secret, still yields a matching public key and still passes a pairwise consistency check. That is C.4.1 example 4, and it is why the RFC specifies bytewise equality. It is also the one corruption that #2709's check_sk + PCT cannot detect, as that PR's call-out notes — so the two mechanisms are complementary rather than redundant.

Expanded key validation. An expanded decapsulation key embeds its own encapsulation key (dk = dk_PKE || ek || H(ek) || z), so the new KEM_KEY_set_raw_expanded_secret_key recovers ek and validates the resulting pair with #2709's KEM_check_key: the FIPS 203 section 7.3 hash check, plus a pairwise consistency test that catches corruption of dk_PKE that the hash check cannot see. Both doors an expanded key can arrive through now go via it — the PKCS#8 expandedKey CHOICE and EVP_PKEY_kem_new_raw_secret_key.

This applies to the encoded private key path only. EVP_PKEY_kem_new_raw_secret_key keeps taking raw bytes at face value, as EVP_PKEY_kem_new_raw_key does — see the note under review considerations for why. Its evp.h contract is updated to say so explicitly, and to route callers who need validation to EVP_PKEY_kem_new_raw_key plus EVP_PKEY_check, or to PKCS#8 parsing.

Keys parsed from both retain their seed, so kem_priv_encode re-serializes them in the seed format section 6 RECOMMENDS. kem_priv_encode itself is unchanged.

Testing

All vectors are from RFC 9935 Appendix C. With this change, every C.4.1 example is rejected, each by the mechanism the RFC describes and with a distinct reason code:

C.4.1 example	Corruption	Rejected by	Reason code
1 (both)	seed and expandedKey disagree	seed consistency check	EVP_R_DECODE_ERROR
2 (expandedKey)	mutated s_0, valid H(ek)	pairwise consistency test	EVP_R_KEM_PCT_FAILED
3 (expandedKey)	mutated H(ek)	FIPS 203 7.3 hash check	EVP_R_INVALID_PRIVATE_KEY
4 (both)	z only; public keys still match	seed consistency check	EVP_R_DECODE_ERROR

New KEMBothFormatTest suite, parameterized over ML-KEM-512/768/1024 with the "Both Format" examples from C.1.1.3, C.1.2.3 and C.1.3.3:

• ParsePrivateKeyBoth — parses, and cross-checks against the other encodings of the same key pair: the expanded secret key matches the C.1.x.2 expandedKey example, the retained seed matches the C.1.x.1 seed example, and the derived public key matches both the seed-only key and the C.2 example public key.
• BothFormatReEncodesAsSeed — re-serializing reproduces the seed-only example byte for byte.
• BothFormatEncapsDecapsRoundTrip — the parsed key is usable for decapsulation.

KEMTest.ParsePrivateKeyBothInconsistent covers C.4.1 examples 1 and 4. Example 4 is the one that pins the bytewise comparison: without it, that key parses successfully.

KEMTest.ParsePrivateKeyExpandedInconsistent covers C.4.1 examples 2 and 3, asserting the distinct reason codes above so the two checks are covered separately rather than as one pass/fail, and asserting rejection through EVP_PKEY_kem_new_raw_secret_key as well as PKCS#8.

KEMTest.ParsePrivateKeyBothInvalidLength covers a truncated both SEQUENCE.

Local results: crypto_test 2933 passed, ssl_test 2579 passed, 0 failures (skips are pre-existing and platform-specific).

Interop testing

Both OpenSSL and BouncyCastle emit the both CHOICE by default, so before this change AWS-LC could not parse any ML-KEM private key either of them produced. Verified by building main + #2709 in a separate worktree and parsing the same files:

	AWS-LC before (main + #2709)	AWS-LC after
OpenSSL 3.6.3 genpkey -algorithm ML-KEM-{512,768,1024}	DECODE_ERROR	parses
BouncyCastle 1.82 KeyPairGenerator("ML-KEM-{512,768,1024}")	DECODE_ERROR	parses

Full matrix against OpenSSL 3.6.3 and BouncyCastle 1.82 (bcprov-jdk18on, JDK 21), for all three parameter sets — all pass:

• peer generates a key (both CHOICE, 1706 / 2474 / 3242-byte privateKey for 512 / 768 / 1024) → AWS-LC parses it;
• the public key AWS-LC derives from the embedded seed equals the one the peer exported separately;
• peer encapsulates → AWS-LC decapsulates → shared secrets match;
• AWS-LC encapsulates → peer decapsulates → shared secrets match;
• AWS-LC re-encodes the key, which emits the seed CHOICE per section 6's RECOMMENDED (66-byte privateKey) → the peer reads it back and still decapsulates to the same secret.

That last point exercises the round trip in both directions: both in, seed out, and the peer accepts the result.

The four Appendix C.4.1 bad keys were also run through the openssl pkey tool built from this branch, confirming end to end (not just in unit tests) that each is rejected with the reason code in the table above.

Review considerations

Validation is scoped to the encoded-key path, and deliberately not to EVP_PKEY_kem_new_raw_secret_key. RFC 9935 section 8 governs parsing the ASN.1 private key, which is what this PR changes; a raw-bytes importer is a different contract, where the caller asserts validity. aws-lc-rs relies on that contract for ML-KEM — DecapsulationKey::new accepts an all-zero decapsulation key of correct length, and aws-lc-rs asserts that a key built from raw private bytes has no encapsulation key — so validating there would be a breaking change for a first-party consumer.

Scoping it this way also costs little in practice. FIPS 203 section 7.3 requires the hash check "before use", and mlkem-native's mlk_kem_dec already performs it at decapsulation time (mlkem/kem.c), so a raw-imported bad key is not silently usable; validating at parse time mainly improves when and how clearly it is reported. The PCT is the only genuinely new check, and FIPS does not require a PCT on imported keys.

Note that EVP_PKEY_check is not usable on a key from EVP_PKEY_kem_new_raw_secret_key: ML-KEM validation compares the two halves of the pair, and that constructor leaves the public half unset, so the call fails with EVP_R_MISSING_PUBLIC_KEY. A caller who wants the full check on raw bytes must supply both halves via EVP_PKEY_kem_new_raw_key. The evp.h comments now say this, and the neighbouring EVP_PKEY_kem_check_key comment — which claimed EVP_PKEY_check accepts a public-only key, when that is EVP_PKEY_public_check — is corrected alongside it.

Behavior change: an expandedKey parsed from PKCS#8 now has a public key. Because ek is recovered from dk, kem_priv_decode case 2 now populates public_key. Such a key previously failed EVP_PKEY_encapsulate outright, so this makes it usable, but it does retire a state the tests covered: KEMTest.PubCmpExpandedPrivateKeyNullPublic becomes PubCmpExpandedPrivateKey and asserts the recovered public key matches the published one. The NULL-public_key branch of kem_pub_cmp is still reachable and still matters — X509_check_private_key reaches it during PKCS12_parse — so the test keeps covering it by clearing the field.

Trailing data after the CHOICE is now rejected, for all three CHOICEs. The CHOICE is the entire contents of the privateKey OCTET STRING, so bytes following it are malformed DER. kem_priv_decode did not verify that key was fully consumed, so such bytes were silently ignored. EVP_parse_private_key rejects trailing data inside the outer PKCS#8 SEQUENCE but never re-examines the privateKey contents after priv_decode returns, so kem_priv_decode is the only place this can be caught.

This tightens the seed and expandedKey cases as well as both. p_kem_asn1.c was the outlier here: ed25519, x25519, dsa, rsa, ec and dh in the same directory all check CBS_len(key) != 0. Well-formed DER is unaffected — the OpenSSL and BouncyCastle keys in the interop matrix above all still parse.

Covered by KEMBothFormatTest.TrailingDataAfterChoiceRejected, which re-encodes each of the three CHOICEs with one extra byte inside the privateKey OCTET STRING and asserts EVP_R_DECODE_ERROR. It asserts the unmodified key parses first, so a failure is attributable to the trailing byte and not to the re-encoding.

Note that pqdsa_priv_decode has the same omission for ML-DSA. That is outside this PR's scope; happy to align the two in a follow-up.

Cost. Parsing an expandedKey private key now runs a hash check plus one encaps/decaps. The hash check is negligible; the PCT is roughly keygen-scale. The seed and both CHOICEs are unaffected, and both was already doing a keygen. Since section 6 RECOMMENDS the seed format, the common path does not change. Raw byte import is unaffected.

FIPS boundary. KEM_KEY_set_raw_expanded_secret_key lives in crypto/fipsmodule/kem/kem.c alongside KEM_check_key. No public API or ABI change. The evp.h contract of EVP_PKEY_kem_new_raw_secret_key is updated to state explicitly that it takes in at face value and leaves the public key unset, and to name the two routes that do validate: EVP_PKEY_kem_new_raw_key followed by EVP_PKEY_check, or parsing the PKCS#8 encoding.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[github-actions]

🔒 Security Review — View Report

Please review before merging.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 87.73842% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.09%. Comparing base (408de5e) to head (9a7e10c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
crypto/fipsmodule/kem/kem.c	77.30%	37 Missing ⚠️
crypto/evp_extra/p_kem_test.cc	95.20%	3 Missing and 3 partials ⚠️
crypto/evp_extra/p_kem_asn1.c	88.23%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3452      +/-   ##
==========================================
+ Coverage   78.06%   78.09%   +0.03%     
==========================================
  Files         700      700              
  Lines      124704   125036     +332     
  Branches    17325    17366      +41     
==========================================
+ Hits        97356    97653     +297     
- Misses      26481    26511      +30     
- Partials      867      872       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.