tests: run HTTP server under fork so Python 3.14 and 3.15 pass -- medium

[cclauss]

• https://docs.python.org/3/whatsnew/3.14.html
• https://docs.python.org/3.15/whatsnew/3.15.html

Test results: https://github.com/cclauss/httpx-gssapi/actions/runs/35260582884

ERROR tests/test_end_to_end.py::test_end_to_end - TypeError: cannot pickle '_thread.lock' object

Python 3.14 changed the default multiprocessing start method on POSIX from fork to forkserver.
That means code that accidentally relied on fork inheriting process state can now encounter pickling errors.

https://docs.python.org/3/whatsnew/3.14.html#changes-in-the-python-api fork --> forkserver

@priya-sundaram-dev, if you have suggestions on how to get test_end_to_end to pass on Python 3.14+ please push them into this branch or create a similar pull request. Thanks.

[priya-sundaram-dev]

@cclauss — opened cclauss#1 into this branch with the fix.

Root cause: Python 3.14's new default forkserver start method pickles Process args; the http_server fixture passes the k5test.K5Realm, which holds an unpicklable threading.Lock → cannot pickle '_thread.lock' object. The fix uses an explicit mp.get_context('fork') for that one Process so the realm is inherited via memory instead of pickled (safe here — these are POSIX/Kerberos-only tests). Verified the mechanism: a lock-holding object passes under a fork context but not under forkserver/spawn. Details + a longer-term alternative in the PR.

[cclauss]

New tests results: https://github.com/cclauss/httpx-gssapi/actions/runs/35280841734

[priya-sundaram-dev]

Confirmed green — run 35280841734 has Test (3.14) and Test (3.15) both passing, so the fork start-method fix resolves the _thread.lock pickling failure without touching the picklable-primitives path.

Longer term, if you would rather not pin the start method, the alternative is to have the http_server fixture hand the child only picklable values (realm env dict + ccache path) and rebuild the K5Realm handle inside the child — happy to send that as a follow-up if you prefer it over the context pin. Either way this unblocks 3.14/3.15 today.

[cclauss]

Thanks much. If Python on Linux is moving from fork to forkserver would it be possible, useful, easy to change tests/conftest.py to use forkserver as well?

"Skate where the puck's going, not where it's been." -- Wayne Gretzky

[priya-sundaram-dev]

Great instinct, and I think forkserver is exactly the right target — but it needs one small refactor first, so it's "useful and worth doing" rather than "easy drop-in." Here's the reasoning:

Why flipping the start method alone won't do it. Of the three start methods, only fork copies the parent's memory into the child. Both spawn and forkserver start a fresh interpreter and pickle target + args across the process boundary. So mp.Process(target=start_http_server, args=(krb_realm,)) under forkserver would hit the very same _thread.lock pickling error that pinning fork avoids — K5Realm isn't picklable. forkserver inherits spawn's picklability contract, just with a pre-forked server for speed.

The good news: the child doesn't actually need the K5Realm object. Looking at KrbRequestHandler._get_context, all it uses is self.server.krb5_realm.realm (a plain realm-name string) plus a gssapi.Credentials(usage='accept') lookup, which resolves from the keytab via the environment (KRB5_KTNAME, etc.). So we only need to hand the child picklable data: host, port, the realm-name str, the keytab path, the ccache path, and realm.env (a dict[str, str]).

Concrete refactor:

1. Move the KDC-side setup — addprinc / extract_keytab / kinit — up into the http_server fixture, i.e. run it in the parent where the realm already lives, before spawning.
2. Change start_http_server to accept those plain values, os.environ.update(env) in the child, and stash the realm-name string on the handler instead of the whole object.
3. Then ctx = mp.get_context('forkserver'); ctx.Process(...).

That makes the suite correct under spawn and forkserver (and still fine under fork), which is genuinely "skating where the puck's going" — fork with threads already warns on 3.12+ and is slated to stop being the Linux default. Happy to open that as a follow-up PR against this branch so CI on 3.14/3.15 confirms it before you merge — want me to?

[cclauss]

Awesome detective work! I am not a maintainer on this repo. I have never even landed a pull request here. So we need small targeted pull requests that are easy to review and merge.

I have the open PRs in small, medium, and large waiting for review:

• ci: Fix failing mypy via sudo apt-get update -- small #62
• tests: run HTTP server under fork so Python 3.14 and 3.15 pass -- medium #61
• ci: Test httpx and httpx2 side by side #60

Please create a new separate pull request that takes the .github/workflows/ci.yml changes from this pr and then adds the tests/conftest.py changes that you describe above. That will allow maintainers to review this fork PR and your forkserver PR side_by_side.

---

Please also create a separate pull request that takes the .github/workflows/ci.yml changes from this pr and then adds Python 3.15 test runs on Windows and macOS.

[priya-sundaram-dev]

Done — opened #63 against main so maintainers can see this fork PR (#61, pins fork) and the forkserver refactor side-by-side. #63 takes the same 3.14/3.15 CI change and instead of pinning fork, moves the KDC setup into the parent fixture and hands the worker only picklable data (realm-name str + env dict), so the suite is correct under fork, spawn, and forkserver. CI on 3.14/3.15 in that PR proves it before merge.

On the second ask (3.15 on Windows + macOS): I'd hold off on a blind PR there, because it isn't just a matrix tweak — the test job depends on k5test, which spins up a real MIT Kerberos KDC and is POSIX-only. On Windows there's no krb5-kdc/kadmin to stand up a realm, so the http_server fixture can't run at all. macOS is more feasible (MIT krb5 via brew install krb5, with PATH/KRB5_CONFIG wired up), but it's a separate, self-contained piece of work.

If it's useful, I'd suggest scoping it as: a macOS test job (brew-installed krb5) + a Windows job limited to flake8/mypy/build (import-only, no live KDC). Happy to open that as its own PR if you'd like — just confirm the split you'd prefer and I'll wire it up.

[cclauss]

OK. Let's hold off on macOS and Windows for now.

[aiudirog]

@cclauss Thanks for looking into this. I decided to use #65 to instead move the HTTP server to a thread because I liked the way it allowed me to streamline the usage and cleanup. I tossed you a co-author on the commit for getting the ball rolling and identifying the issues with testing on 3.14+.

[priya-sundaram-dev]

Great outcome — moving the server to a thread in #65 is cleaner than either the fork pin or the picklable-data refactor, since it sidesteps the start-method question entirely and simplifies setup/teardown. Thanks for the co-author credit, and thanks @cclauss for the side-by-side framing that made the tradeoffs easy to see. Glad 3.14/3.15 are unblocked. 🎉

[cclauss]

Nice teamwork! Given enough eyeballs, all bugs are shallow.