Semantic embedding client uses static webpki-roots — can't trust corporate/internal CA, ignores SSL_CERT_FILE

[akumria]

Short description

Semantic index build fails with a TLS UnknownIssuer error against an embedding endpoint served behind a private/corporate CA; rustls client trusts only bundled Mozilla roots and ignores SSL_CERT_FILE, with no config/env override.

What happened?

Summary

AFT's semantic-index embedding HTTPS client is built on hyper-rustls with the static Mozilla webpki-roots bundle as its only trust-anchor set. On any network where the configured embedding endpoint is served behind a certificate issued by a private/corporate CA (a TLS-intercepting proxy, or an internal gateway with an org-issued cert), the TLS handshake fails with an UnknownIssuer error and semantic indexing never starts.

Critically, there is no supported escape hatch: no config option, no env var, and the standard SSL_CERT_FILE / SSL_CERT_DIR / NODE_EXTRA_CA_CERTS variables are all ignored by the rustls client.

Result — a permanently degraded state, retrying forever:

semantic index build: embedding backend unavailable
(openai compatible request failed: error sending request for url
 (https://<endpoint>/v1/embeddings)); retrying in 60s

and aft_search reports Semantic search is not enabled / Search status: fully degraded.

Expected

The embedding client honors the OS/system trust store (or an explicit CA bundle), so an endpoint whose cert is already trusted by the machine works — as it does for curl, the browser, and Node-based tooling.

TLS stack (from strings on the binary)

• hyper-rustls 0.27.7
• rustls 0.23.37
• rustls-webpki 0.103.10
• rustls-pki-types 1.14.0, tokio-rustls 0.26.4

Root cause

The client trusts only the compiled-in Mozilla root bundle. Evidence from strings:

• Public roots ARE baked in: ISRG Root X1/X2, Amazon Root CA 1..4, DigiCert Global Root CA/G2, USERTrust RSA/ECC — i.e. the webpki-roots static set.
• Absent: no rustls-native-certs, no rustls-platform-verifier (nothing that consults the OS trust store / macOS keychain).

The internal gateway's leaf chains to a private issuing CA that is present in the OS keychain but not in the Mozilla bundle, so rustls returns UnknownIssuer and the request never leaves the client.

Reproduction

1. Point semantic.base_url at an HTTPS endpoint whose cert chains to a CA that is in the OS trust store but not in the Mozilla webpki-roots bundle (corporate TLS-intercepting proxy, or an internal gateway with an org-issued cert).
2. Provide a valid api_key_env.
3. Open any project; wait for the semantic index build.
4. Observe the repeating embedding backend unavailable ... error sending request log line, and aft_search reporting semantic disabled.

Config used:

"semantic": {
  "backend": "openai_compatible",
  "base_url": "https://<internal-gateway>/v1",
  "api_key_env": "PORTKEY_API_KEY",
  "model": "<titan-embed-v2>",
  "timeout_ms": 25000,
  "max_batch_size": 64
}

Proof it is CA trust, not network / proxy / key

All verified on the affected machine:

• Endpoint is healthy. curl to the exact /v1/embeddings URL, both direct and forced through the corporate proxy, returns a valid 1024-dim embedding vector. Host reachable on both paths; API key valid.
• A correct CA bundle exists and is sufficient. openssl s_client -connect <endpoint>:443 -CAfile <corp-bundle>.pem returns Verify return code: 0 (ok).
• AFT already has that bundle in its environment and still fails. The running daemon's env contains SSL_CERT_FILE=<corp-bundle>.pem and NODE_EXTRA_CA_CERTS=<corp-bundle>.pem, yet AFT still logs the UnknownIssuer-class failure every 60s. Direct proof the rustls client ignores SSL_CERT_FILE and uses only its static bundle.

The only differentiator between "works" (curl/openssl — system keychain or explicit -CAfile) and "fails" (AFT) is the trust-anchor source.

Impact

• Semantic search is completely unavailable for all users on corporate networks with TLS interception, and for anyone whose embedding endpoint sits behind a private CA — regardless of correct config, valid key, and reachable network.
• Confusing to diagnose: the endpoint works from every other tool on the machine (curl, browser, Node tools that honor NODE_EXTRA_CA_CERTS), so users blame the gateway or key rather than AFT's trust store.
• Affects every AFT daemon/session on the host simultaneously (systemic, not per-project).

Requested fix (any one unblocks; ordered by preference)

1. Use the OS trust store — build the embedding client with rustls-platform-verifier (preferred) or rustls-native-certs, so the macOS keychain / Windows cert store / Linux /etc/ssl roots are honored. Zero per-user config on managed machines where the CA is already OS-trusted.
2. Honor SSL_CERT_FILE / SSL_CERT_DIR in the embedding client. Matches curl/OpenSSL convention; least-surprising env override.
3. Add an explicit config option, e.g. "semantic": { "ca_bundle": "/abs/path/corp-certs.pem" }, loaded into the RootCertStore in addition to the bundled roots.
4. (Opt-in escape hatch) "semantic": { "tls_insecure_skip_verify": true } / danger_accept_invalid_certs for constrained environments — useful as a stopgap even if 1–3 land.

Option 1 is the most robust.

Nice-to-have: better error surface

Propagate the underlying rustls cause (UnknownIssuer / invalid peer certificate) into the log line instead of the generic error sending request for url, and have aft_search hint "TLS trust failure to embedding endpoint" rather than a bare "Semantic search is not enabled".

Diagnostics

Plugin version

No response

AFT binary version

0.48.0 (also reproduced on 0.47.3)

Platform

macOS arm64 (macOS 26.5.2, build 25F84)

Log output (optional)