Fix AES-CBC with mbedTLS 3.x: restore PKCS7 padding and apply caller IV

[yveshughes]

Problem

Building with USE_MBEDTLS=ON against mbedTLS 3.x breaks both AES-CBC uses of PlatformCrypto.c, while the same code works with mbedTLS 2.x and OpenSSL. There are two separate divergences from the OpenSSL path:

1. mbedTLS 3.x removed the default PKCS7 padding mode for CBC contexts

In mbedTLS 2.x, mbedtls_cipher_setup() initialized CBC contexts with PKCS7 padding by default (matching OpenSSL's EVP layer, which also defaults to PKCS7). In mbedTLS 3.x this default was removed — a freshly set-up CBC context has no padding mode, and mbedtls_cipher_finish() fails with MBEDTLS_ERR_CIPHER_BAD_INPUT_DATA (-0x6100) on any input.

Since PltDecryptMessage() never set a padding mode explicitly, every ALGORITHM_AES_CBC + CIPHER_FLAG_FINISH operation fails. The user-visible symptom is Failed to decrypt audio packet on every audio packet when streaming from Sunshine or GameStream with an mbedTLS-3.x-built client — audio is completely broken.

Fix: explicitly set MBEDTLS_PADDING_PKCS7 on CBC contexts at first-time init in both PltEncryptMessage() and PltDecryptMessage(), guarded by #ifdef MBEDTLS_CIPHER_MODE_WITH_PADDING.

2. The caller's IV is never applied without CIPHER_FLAG_RESET_IV

The OpenSSL path passes the caller's IV into the first EVP_EncryptInit_ex()/EVP_DecryptInit_ex() call even when CIPHER_FLAG_RESET_IV is not set. The mbedTLS path only calls mbedtls_cipher_set_iv() when that flag is present, so without it the context keeps the all-zero IV from setup and the first block is en/decrypted with the wrong IV.

This affects the pre-Gen 7 CBC remote input encryption in InputStream.c, which initializes the context once with the real IV and then relies on cross-packet CBC chaining (no CIPHER_FLAG_RESET_IV).

Fix: apply the caller's IV (set_iv + reset) during first-time initialization for non-GCM contexts. GCM is excluded because it passes its per-message IV directly to mbedtls_cipher_auth_{encrypt,decrypt}_ext().

Verification

Tested on Linux against mbedTLS 3.6.2 with standalone host-side tests cross-checking against OpenSSL:

• AES-128-CBC/PKCS7 packets encrypted through the OpenSSL PltEncryptMessage() path (Sunshine's audio packet format) decrypt correctly through the fixed mbedTLS PltDecryptMessage() path.
• mbedTLS PltEncryptMessage() CBC output is byte-identical to OpenSSL output across a sequence of packets on one context, confirming the cross-packet CBC chaining behavior (input-stream style, no RESET_IV) matches.

GCM paths (video/control, Gen 7+ input) are unaffected: they always pass CIPHER_FLAG_RESET_IV or use the auth-crypt entry points, and padding does not apply to GCM.

Related: #75 migrated to the mbedTLS 3.x API surface but didn't cover this behavioral change in mbedtls_cipher_setup().

🤖 Generated with Claude Code

[trexx]

Ran in to this whilst updating moonlight-android.