Avoid forking a multi-threaded parent in variogram sampling - #993
Avoid forking a multi-threaded parent in variogram sampling#993manduinca wants to merge 2 commits into
Conversation
sample_empirical_variogram built its pool with the default start method, which forks on Linux and raises a DeprecationWarning when the parent has live threads. Use forkserver where available and spawn otherwise, so the workers start from a process without them.
|
Thanks @manduinca, this warning will stop polluting the outputs. A note: The |
|
Thanks @rhugonnet, good to know about the revamp. One thing worth flagging: the 15 failures the CI reports here are not from this branch. They are all in The |
|
@manduinca Thanks, indeed it is a separate issue that @marinebcht is fixing in #994. Then we should be able to merge here as well 😉 |
2 participants
Fixes #507
sample_empirical_variogrambuilt its pool withmp.Pool(...), which uses the default start method. On Linux that isfork, and forking a parent with live threads is what raises theDeprecationWarningin the issue. This usesforkserverwhere it exists andspawnotherwise, as suggested in the issue, so workers start from a process without threads.Two things you should weigh, since they are not free:
maxtasksperchild=1recycles a worker after every variogram, so the method matters:forkserverforks from a pre-imported process and stays cheap, whilespawnre-imports xdem each time. That is why the choice prefersforkserverrather than going straight tospawn.fork, callingsample_empirical_variogram(n_jobs>1)from a plain script works; withforkserver/spawnthe caller needs theif __name__ == "__main__":guard. Python 3.14 already changed the default on Linux, so this is where things are heading anyway, but it will affect scripts written against older behaviour.On the test: it holds a thread open while sampling and turns the fork
DeprecationWarninginto an error, which reproduces the issue's conditions. I could not watch it fail locally — macOS already defaults tospawn, so the warning never appears here — but it should cover the regression on the Linux CI.