FEAT: extend profiling to child processes - #431
Conversation
TTsangSC
changed the title
|
Did some more tests on local post-#428-merge, maybe it is just legacy Python and dependency versions causing the issues. Will just rebase, force-push, and see what happens. |
f9a37af to
aca4e2c
Compare
|
Unfortunately there's too little context to determine why the tests are failing on other platforms. Heck I can't even replicate the macOS failures on my machine with matching dep versions. Just wrote in more code for extracting the debug outputs, force-pushed, and hopefully I will have more clues for what to work on. |
|
Added a ton of logging/debug messages, a few debug-only config options and a
Footnotes
|
|
WRT to the existing multithreaded case, IIRC the main point of that is just to ensure we don't hang. I could be misremembering. For the forkserver issue, from what I understand that's a long lived process, so maybe some forkerserve state debugging (not sure if your code does this or not, I haven't looked at it yet) def debug_forkserver_state(label):
import os
from multiprocessing import forkserver
fs = forkserver._forkserver # private CPython state
pid = getattr(fs, "_forkserver_pid", None)
# "Did *this process* already know about / launch a forkserver?"
known_forkserver_pid = pid
known_forkserver_started = pid is not None
# "Was *this current process* itself started by a forkserver?"
started_by_forkserver = forkserver.get_inherited_fds() is not None
# Best-effort liveness check for the known forkserver PID, if any.
# This is optional, and mostly useful in the parent/originating process.
forkserver_pid_alive = None
if pid is not None:
try:
os.kill(pid, 0)
except OSError:
forkserver_pid_alive = False
else:
forkserver_pid_alive = True
print(
f"[{label}] "
f"pid={os.getpid()} ppid={os.getppid()} "
f"known_forkserver_pid={known_forkserver_pid!r} "
f"known_forkserver_started={known_forkserver_started} "
f"forkserver_pid_alive={forkserver_pid_alive} "
f"started_by_forkserver={started_by_forkserver}"
)And maybe also check if the main module attributes differ in any meaningful way? getattr(sys.modules['__main__'].__spec__, 'name', None)
getattr(sys.modules['__main__'], '__file__', None)
multiprocessing.get_start_method()But IDK, your guess is probably better than mine at this point. |
|
Thanks for the input! May have to take a closer look at the I have a piece of good news and a half:
Footnotes
|
|
Figured it out, the issue is that:
I think the bug is fixed (will push shortly), but I've noticed significant performance regression (about 100% slowdown of |
|
Tests seem flaky... there is no good reason that 6e60a6a failed on building for |
TTsangSC
changed the title
|
Hi @Erotemic, I think this is ready for review if you have the time... sorry for the metric ton of code. |
Erotemic
left a comment
Erotemic
left a comment
There was a problem hiding this comment.
It will take me a while to fully go through this, but here is a start.
| ... print("We probably didn't count up to 100 but whatever") | ||
| We probably didn't count up to 100 but whatever | ||
|
|
||
| >>> with ( # doctest: +NORMALIZE_WHITESPACE |
There was a problem hiding this comment.
Note, because we use xdoctest, you should be able to put # doctest: +NORMALIZE_WHITESPACE as a standalone comment above the code for it to apply to every line after. Although I don't often use NORMALIZE_WHITESPACE, so I can't say I'm 100% confident in that.
| dir=get_path('purelib'), | ||
| ) | ||
| try: | ||
| pth_content = 'import {0}; {0}.load_pth_hook({1})'.format( |
There was a problem hiding this comment.
This has the potential to cause some import overhead. IIUC, we are going to effectively import the entire line_profiler.init. I'm not sure if that's going to be noticeable or not. One idea is to make a separate _line_profiler_hooks package that's just a minimal top level module for very fast import, but that may be overengineering. I'd need to think about it and probably test it.
There was a problem hiding this comment.
One data point on my machine, though I think other platforms are most likely gonna behave the same: importing line_profiler._child_process_profiling.pth_hook causes the following to be imported:
- All top-level
line_profilersubmodules except the following:~.cleanup~.curated_profiling~._threading_patches~.ipython_extension
- None of the
line_profiler.autoprofilecomponents, and - None of the
line_profiler._child_process_profilingcomponents except~~.pth_hookitself.
So, as one would intuit, most of the "core" stuff directly used by line_profiler.LineProfiler and @line_profiler.profile. Could be worthwhile to set it up as a separate namespace... and it should be trivial to configure, given that we already create two separate namespaces (line_profiler and kernprof).
The remaining question is more of a design one: should write_pth_hook() stay in said namespace for symmetry with load_pth_hook(), or should it become an instance method of LineProfilingCache's?
| ) | ||
| if not fnames: | ||
| return LineStats.get_empty_instance() | ||
| return LineStats.from_files(*fnames, on_defective='ignore') |
There was a problem hiding this comment.
Maybe on_defective='warn' makes more sense here?
There was a problem hiding this comment.
Could be. The original motivation is that:
- Since the
._setup_in_child_process()code is run in all child processes at startup incl. unused worker processes and the fork-server process, each such process occupies and secures a tempfile name (*.lprof) by touching it. Said tempfile is supposed to be written to when the corresponding Python process terminates. - But then some of these files could result in errors upon reading: processes killed via signals before cleanup is run result in empty tempfiles, and those killed mid-cleanup make for corrupted ones.
- I opted to suppress the error/warning messages given that they originate directly from
pickleand are thus not the most intuitive/helpful.
But maybe that's just indication that we should've vet the input files better before passing them onto pickle. And given that the rest of the PR has seen a lot of iteration after these lines were first written, we should probably get a lot fewer false alarms from the warnings, and where we do have something to warn it will be of actual problematic cases (e.g. unclean exits resulting in incomplete data).
| """ | ||
| # XXX: why can `coverage` get away with not doing all these | ||
| # lock-file hijinks and just patching `BaseProcess._bootstrap()`? | ||
| def get_poller_args( |
There was a problem hiding this comment.
Do these need to be nested functions? Would a helper class make more sense than relying on closures / be more testable?
Do we have a test for a worker that does some non trivial processing and then intentionally terminates so we can have an explicit comparison between the unpatched and patched terminate and make sure they both behave similar enough?
There was a problem hiding this comment.
Yeah I should probably refactor wrap_terminate() for readability. We've barely used closure variables in the nested funcs anyway (just cache in process_has_returned()); still, that can be easily remedied, given that I can just pass cache to the function in the _Poller.poll_until() constructor.
We do test Process.terminate() as shown in the coverage report, albeit indirectly:
- In normal ("happy-path") code execution
Process.terminate()isn't even called – AFAIK it only happens when the parallel workload errored out, which for whatever reason may leave the child process stuck "in limbo", where e.g.atexitcleanup callbacks may or may not be executed, or be in various stages thereof. - Such children are
.terminate()-d as a part of finalization of the parentPool(seePool.__init__()and.terminate()), which in particular does not wait for child-process cleanup, thus necessitating the patch. - In order to test this we have tests1 where the parallel workload can be "set up for failure". Those are where the coverage on
wrap_terminate()comes from. A quick check is how running the test suite with-k "test_child and success"only gives 52% coverage onmultiprocessing_patches.pywhile-k "test_child and failure"gives 72%.
Given that the existing code do call vanilla_impl() in a finally clause, it is probably the closest we can get to the unpatched behavior in that we (1) do eventually send the SIGTERM, and (2) before that, give child processes (as far as possible – which isn't 100% of the time, hence the necessity of the timeout) the required grace to run cleanup and write profiling data. Anyhow, maybe we can look into tests where we explicitly create a Process object and later .terminate() it, if that would be more assuring.
Footnotes
-
Granted, we just sum over consecutive integers in those tests. They have the benefit that the workload size corresponds directly to the num-hits on the loop line, and hence we can keep track of whether the gathered profiling data are complete. If that's too trivial maybe we can do Fibonacci or something instead. ↩
| self, obj: Any, attr: str, value: Any, *, | ||
| name: str | None = None, | ||
| cleanup: bool = True, | ||
| priority: float = 0, |
There was a problem hiding this comment.
Thanks, good catch!
| ) | ||
|
|
||
|
|
||
| @pytest.mark.retry(_NUM_RETRIES, |
There was a problem hiding this comment.
Why is a retry necessary here? Is there something non deterministic going on? Any chance we are hiding a race condition? Similar question with other pytest.mark.retry marked tests.
There was a problem hiding this comment.
The short answer
Yes, workers with failing workload ends up in limbo sometimes, as mentioned in the other comment. It should theoretically be possible to drop the retry on POSIX (and even the whole wrap_terminate() deal) because of the use of SIGTERM handlers, but the consensus seems to be that signal handling is busted on Windows, so we still need the timeout which is by nature flaky.
The long answer
- Again, processes receiving parallel workloads which don't error out seem to always exit cleanly and never need to be
.terminate()-d. Those which don't... don't. Which sets up the potential race condition: the failing child could be cleaning up, stuck in limbo (possibly waiting for communication from the parent?), or whatever; meanwhile the parent don't really care either way, because it already received all the results (i.e. pickled return-value objects or exceptions) from the children, and is ready to.terminate()whichever child process that remain alive. - In earlier iterations of the PR I just set each child process up to manage a lock file, and had the parent process wait without a timeout for said file before sending the
SIGTERM(which is essentially the only thing that the vanillaProcess.terminate()does). Which then caused the tests to sometimes hang indefinitely, meaning that in some failing child processes we never naturally got to the point where thefinally: cache.cleanup()clause is executed. Unfortunately such failure also seems to be nondeterministic, at least at a cursory glance. - Hence the timeout. Which did alleviate the issue in that we do get all of the profiling data most of the time – because the parent process no longer willy-nilly kills the children, at least until the timeout expires. This handles cases where the child process is healthy enough to actually attempt cleanup, but would have been prevented from doing so by the parent's
.terminate()-ing it. - But then of course there are still corner cases for when the children is FUBAR – and I think it can be argued that those cases are what the
Process.terminate()method is for in the first place. Like, in a perfect would the child process would've just executed itsatexithooks and exited non-zero, right after raising the error and pushing it to the communication queue with the parent. But obviously that doesn't always happen. In those cases anything goes... hence the retries.
Notably, coverage also seems to be having trouble with consistently handling data collection when child processes are .terminate()-d, and just gives up while on Windows:
@pytest.mark.skipif(env.WINDOWS, reason="SIGTERM doesn't work the same on Windows")
@pytest.mark.flaky(max_runs=3) # Sometimes a test fails due to inherent randomness. Try more times.
class SigtermTest(CoverageTest):What do we do?
I didn't like having to retry either – but here we are. But maybe we can at least refactor and separate the tests, so that we make sure that the mark is only applied to cases where it might be needed/justified (i.e. tests with failing parallel workloads and on Windows).
That or we just skip (as coverage does) or XFAIL on Windows.
|
I have a new idea for improving determinism in profiling
However, this strat does have complications:
|
|
As it is now, the code (at least on non-Windows platforms) already reliably and deterministically captures all profiling data related to tasks set to the child processes. If the child process gets to execute any task at all, it must have gone through setup which allows for the capturing of profiling data. What however is more difficult to handle consistently, and is causing most of the recent pipeline failures (other than 25785347402, which was entirely on me), is what happens with child processes which terminates without ever having received a task.1 While currently the Such failure calls into question:
Footnotes
|
|
Added the PID bookkeeping, but there seems to be an unrelated Heisenbug which I'm struggling to pin down. Every ≈ 1000–2000 runs of The newest pipeline failures are semi-related:
|
|
@Erotemic yikes, unfortunately job 76507649976 seems to have gotten stuck despite multiple precautions (timeouts for dubious sections, hundreds/thousands of rounds of local tests)... worse, its the Mac job that's stuck (which I suppose is the most expensive on CI), despite that being the most thoroughly tested env, since it's also my local. I don't have the perms to cancel the job; please do so ASAP before it continues to rack up compute. Terribly sorry for this. |
|
Yikes! Just saw the message. I don't see an immediate way to cancel it. Maybe it timed out and it just looks like it is churning? It does say: "The job has exceeded the maximum execution time of 6h0m0s", and in this PR I see: Cancelled after 365m. So I think it is not running. |
|
Yep, I think Actions have a default 6 h limit (not sure if it's the entire pipeline or individual jobs) on GitHub. This happened on Monday so said default timeout expired a while before intervention... it may not be ideal, but we can probably put in loose-ish timeouts for For the (probably1) offending test We can probably mitigate this by folding the cases tested by Footnotes
|
|
When trying to sniff around and trigger and fix the bug I ran into an even weirder issue.
This seems to indicate something being wrong when profiling starts on a non-main thread; in view of that, how |
|
Sorry for the lack of update. Was a bit shellshocked after the last blunder and held off on making pushes until I can diagnose what happened and better reproduce the sporadic hanging bug... ... in which however I am not quite successful. It still only shows up every ≈1000 tests or so, and apparently according to the logs, that is due to the signal handler's somehow not firing on one of the child processes; still trying to figure out how that is at all possible. Still I've made some further changes:
If it isn't as safe and deterministic to temper with signal handling as I've initially hoped, we may just have to report profiling stats more granularly like on Windows, where each multiproc pool task prompts the child process to write the profiling stats to its assigned file. Since one of the patches already patches task/result queues to send the child-process identity back to the parent alongside the task result, it should be easy (though with overhead) to also slip the collected stats in. Will see if that helps. Oh and looks like I broke something else completely related... not a good sign. The lint-job failure seems unrelated though, it was on |
|
Still working on this on-and-off, but I'm unfortunately still stuck with flaky tests... seems that there's something which makes SIGTERM handling inherently unreliable in Unfortunately this means that there's no guarantee for profiling output when a child process is |
fe70770 to
bc3adb9
Compare
- `line_profiler/curated_profiling.py`
New module for setting up profiling in a curated environment
- `ClassifiedPreimportTargets.from_targets()`
Method for creating a `ClassifiedPreimportTargets` instance,
facilitating writing pre-import modules in a replicable and portable
manner
- `ClassifiedPreimportTargets.write_preimport_module()`
Method for writing a pre-import module based on an instance;
also fixed bug where the body of the written module was intercepted
without appearing in the debug output
- `kernprof.py`
- `_gather_preimport_targets()`
Migrated to `line_profiler.curated_profiling`
- `_write_preimports()`
Now using the new `ClassifiedPreimportTargets` class, moving esp.
the logic to the `write_preimport_module()` method
- `kernprof.py::_manage_profiler` `line_profiler/curated_profiling.py::CuratedProfilerContext` New context-manager classes for handling profiler setup and teardown - `kernprof.py::_pre_profile()` Refactored into the above context managers and other private functions (`_prepare_profiler()`, `_prepare_exec_script()`)
line_profiler/_child_process_profiling/cache.py::LineProfilingCache
New class for passing info onto child processes so that profiling
can resume there
line_profiler/pth_hook.py
New submodule for the .pth-file-based solution to propagating
profiling into child processes:
write_pth_hook()
In the main process, write the temporary .pth file to be loaded
in child processes
load_pth_hook()
Called by the .pth in child process, loading the cache and
setting up profiling based thereon
line_profiler/_child_process_profiling/cache.py::LineProfilingCache
Added new `.profile_imports` attribute to correspond to `kernprof`'s
`--prof-imports` flag
line_profiler/_child_process_profiling/meta_path_finder.py
New submodule defining the `RewritingFinder` class, a meta path
finder which rewrites a single module on import
line_profiler/_child_process_profiling/pth_hook.py
write_pth_hook()
Now also handling the `os.fork()` patching/wrapping
_setup_in_child_process()
Now creating a `RewritingFinder` to mirror what
`~.autoprofile.autoprofile.run()` does in the main process
.
(See `fable-review-pr431-2026-07-05.md::P13` in #5) line_profiler/_child_process_profiling/_cache_logging.py TIMESTAMP_PATTERN Added formatting field `level` add_timestamp() Added optional argument `level` CacheLoggingEntry .__doc__ Updated instantiations in doctest .level New field (2nd item) .to_text(), .from_text(), .write() Added handling of the `.level` field .new() Added optional argument `level` line_profiler/_child_process_profiling/cache.py::LineProfilingCache _debug_output() - Added optional argument `level` in keeping with the base class - Moved implementation of `._make_debug_entry` (used nowhere else) inside _setup_in_child_process() Updated patching of `CuratedProfilerContext._debug_output` _make_debug_entry Removed line_profiler/cleanup.py::Cleanup _debug_output() Added optional argument `level` for setting the logging level _cleanup() If a callback fails, it is now logged with `Logger.warning()` instead of `Logger.debug()`
(See `fable-review-pr431-2026-07-05.md::P10` in #5) line_profiler/line_profiler.py::LineStats.from_files() - Added new private params `_note_on_empty` and `_note_on_defective` so that calling methods can bolster the emitted warning messages with relevant notes - Improved formatting for errors without arguments in warnings - Fixed formatting of the empty-file warning messages - Reworded defective-file warning messages line_profiler/_child_process_profiling/cache.py ::LineProfilingCache.gather_stats() Now calling `LineStats.from_files()` with notes for more helpful warning messages
(See `fable-review-pr431-2026-07-05.md::P3` and `P12` in #5) line_profiler/_child_process_profiling/_retrieve_pids.py New helper submodule for checking which PIDs are active Caveats: - There is no Python-native method for checking the process tables so the implementations are platform-specific and relies on parsing command output (`ps` on POSIX, `tasklist` on Windows) - Said parsing esp. on Windows may be locale-sensitive - TODO: we haven't even tested whether Windows implemention works yet... line_profiler/_child_process_profiling/cache.py::LineProfilingCache _setup_in_main_process() Now allowing for `.write_pth_hook()` to error out, emitting a warning that profiling cannot continue in children except for forked ones write_pth_hook() - Added param `clean_stale` to optionally run cleanup for .pth files associated with stale main PIDs; during normal execution they should've been cleaned up by the owning process, but if it is e.g. killed this may be necessary to prevent .pth files from piling up - Now including info about the main PID in the filename of the .pth file, along with other normalizations of the affixes; note that this means the filename stem is no longer guaranteed to start with `prefix` and end with `suffix` - Added search order for the default `dir` to write the .pth file to: - Location of `_line_profiler_hooks` if a `site-packages` directory - `sysconfig.get_path('purelib')` (existing behavior) - `site.getusersitepackages()` if `site.ENABLE_USER_SITE` is true The .pth file is written to the first writable location - The .pth file written now fails gracefully in case the function (`_line_profiler_hooks.load_pth_hook()`) to be imported and called cannot be found for whatever reason _cleanup_stale_pth_files(), _find_pth_files() New helper methods for locating and pruning .pth files associated with stale PIDs _get_graceful_oneline_call() New (doctest-ed) helper method for writing a .pth file calling a function, which gracefully/silently fails in case of module/function lookup errors; this prevents .pth files installed to shared locations from tripping up other interpreters, esp. those without `_line_profiler_hooks` installed _enumerate_pth_installation_locations() New helper method for enumerating the possible locations to install a .pth file to line_profiler/rc/line_profiler.toml ::[tool.line_profiler.child_processes.pth_files] Updated docs
Forked children reuse the parent's profiler object (necessary: it holds the already-wrapped functions), so their stats dumps contained all data the parent had accumulated before the fork. The parent dumps that same data itself, and gather_stats() merges by summation, so every line executed before a fork was counted once per forked child: a 100k-loop parent with three fork children reported 400,004 hits instead of 100,001, for hits and times alike. fork is the default start method on Linux <= 3.13, so merged --prof-child-procs numbers were wrong in the most common configuration. (Evidence: dev/planning/ fable-review-pr431-2026-07-05.md, finding P1.) Fix: snapshot the profiler's stats at the top of the os.fork wrapper's child branch and pass them down to _DumpStatsHelper as a subtractive baseline; every dump now writes get_stats() - baseline via the new LineStats.__sub__/__isub__ (the exact inverse of __add__, raising ValueError when the subtrahend is not a prefix of the minuend). The delta is re-derived on every dump since pool workers dump per task. Nested forks fall out naturally: each fork snapshots the state at its own fork point. New tests: tests/test_child_procs/test_fork_stats_attribution.py runs the real CLI in a real subprocess over fork/spawn/forkserver x Process/Pool/ProcessPoolExecutor and asserts exact hit counts, plus a multi-fork variant asserting counts do not scale with child count. (These also add the first ProcessPoolExecutor coverage.) LineStats subtraction is covered by doctests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mandatory pool patch changes the parent<->worker result protocol (workers push their PID with each result), and the patched parent unconditionally 2-way-unpacked every result. A worker that was never patched -- e.g. its interpreter never loaded the profiling .pth hook, or multiprocessing.set_executable() points at a different Python -- pushes vanilla 3-tuples, so the unpack raised ValueError inside the pool's result-handler thread; the thread died and every AsyncResult.get() blocked forever. --prof-child-procs turned a working program into a deadlock (reproduced; evidence: dev/planning/ fable-review-pr431-2026-07-05.md, finding P2), and load_pth_hook swallowing all exceptions made every silent hook failure take this path. Fix: - Tagged protocol: patched workers now push (PID_TAG, pid, obj) triplets, where PID_TAG is a distinctive module constant. - Tolerant parent: _wrap_outqueue_quick_get() strips tagged results and passes anything else through untouched with a once-per-session UserWarning that the worker's profile data will be missing. - Loud hook failure: load_pth_hook() now always warns when child setup fails (previously only under DEBUG) while still letting the child run unprofiled. Tests: unit tests for the tagging/unwrapping, an integration test running a patched parent against genuinely vanilla spawn workers (hard-timeout-guarded: a regression fails in seconds instead of hanging CI), and a fork control case asserting no warning when the workers are patched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
line_profiler/_child_process_profiling/multiprocessing_patches
/_mandatory_patches.py
_LOCK_FILE_LOC
Removed unused constant
wrap_handle_results(), wrap_worker()
No longer exposed (now automatically generated from more basic
callbacks)
_wrap_outqueue_quick_get(), _warn_untagged_result_once()
Refactored into `._queue.QuickGetWrapper`
POOL_WORKER_PID_PATCH
Now using the new `get_mp_pool_patch()` to generate the patching
wrappers for `Pool._handle_results()` and `worker()`
line_profiler/_child_process_profiling/multiprocessing_patches
/_queue.py
PID_TAG
Removed
PutWrapper.__init__()
Updated call signature: instead of `push_to_parent`, it now
takes an optional `tag: str | None = None`, where the callback's
return value is `.put()` along with the provided `tag` where
available
QuickGetWrapper
New helper object generalizing the old
`._mandatory_patches._wrap_outqueue_quick_get()`
get_mp_pool_patch()
New helper function for generating the patch for
`multiprocessing.pool`, pairing up the callbacks which resp.
returns the extra data in the child process and processes said
data in the parent
tests/test_child_procs/test_pool_protocol.py
<General>
Added type annotations
test_put_wrapper_tags_results()
Refactored because `PID_TAG` no longer exists
test_quick_get_wrapper_unwrapping_results()
- Refactored from the old `test_quick_get_unwrapping()`, because
(1) `_wrap_outqueue_quick_get()` no longer exists, and (2)
`QuickGetWrapper` allows for more directly testing the desired
behavior
- Added missing check against repeated warning
- Added check for the data-processing callback
test_patched_parent_with_vanilla_spawn_worker()
Updated warning check because of the changed warning messgae
emitted
(See `fable-pr431-plan-2026-07-05.md::P8` in #5) line_profiler/_child_process_profiling/multiprocessing_patches /__init__.py _PATCHES Now loaded from `Registry.get_default()` apply() Replaced assertion with `RuntimeError` line_profiler/_child_process_profiling/multiprocessing_patches /_infrastructure.py::Registry DEFAULT_ENTRY_POINT, from_entry_point() Removed (we don't actually expect to load patches from 3rd party packages, so just hard-code the patch locations) get_default() New instantiator refactored from `.from_entry_point()` loading patches from the hard-coded sibling submodules setup.py Removed the now-unused `line_profiler._multiproc_patches` entry point
(See `fable-review-pr431-2026-07-05.md::P4` in #5) line_profiler/_child_process_profiling/multiprocessing_patches /_mandatory_patches.py wrap_join_exited_workers() New wrapper around `Pool._join_exited_workers()` which mirrors the bookkeeping that `wrap_terminate_pool()` does for workers, except instead it is for that exited early during the pool's lifetime wrap_terminate_pool() Replaced assertion in the `finally` clause with a warning that some worker processes remains alive after calling `Pool.terminate()` RebootForkserverPatch.reboot() Rewrote bare assertion to be more informative (shouldn't happen though)
(See `fable-pr431-plan-2026-07-05.md::P11` in #5) line_profiler/_child_process_profiling/cache.py _StatsHelper - Refactored from `_DumpStatsHelper` - Added new methods `.get()` and `.dump()`, paralleling `LineProfiler.get_stats()` and `.dump_stats()` LineProfilingCache._stats_helper - Renamed from `._stats_dumper` - Now a `_StatsHelper | None` line_profiler/_child_process_profiling/multiprocessing_patches /__init__.py::apply.__doc__ line_profiler/rc/line_profiler.toml ::[tool.line_profiler.child_processes.multiprocessing.patches] Updated descriptions of the `pool` patch line_profiler/_child_process_profiling/multiprocessing_patches /_mandatory_patches.py wrap_terminate_pool(), wrap_join_exited_workers() Removed (refactored into `.._pool_patch_helpers.get_worker_finalization_patch()`) POOL_WORKER_PID_PATCH - Updated data-processing callback to be compatible with `get_per_task_callback_patch()`, and to emit debug-log messages for the # tasks handled by the worker - Now also constructed with `get_worker_finalization_patch()` line_profiler/_child_process_profiling/multiprocessing_patches /_pool_patch_helpers.py get_per_task_callback_patch() - Refactored from `.._queue.get_mp_pool_patch()` - Updarted call signature of `get_data`: `(LineProfilingCache) -> T` - Added optional param `patch` so that we can stack the `get_*_patch()` functions get_worker_finalization_patch() New helper function Rrefactored from `.._mandatory_patches.wrap_terminate_pool()` and `.wrap_join_exited_workers()`, returning a single `SingleModulePatch` object patching the resp. methods to execute callbacks on finalized worker processes line_profiler/_child_process_profiling/multiprocessing_patches /_profiling_patches.py wrap_worker() Removed POOL_PATCH - Reworked functionality: instead of writing the accumulated stats to disk on each task processed, worker child processes now send the stats back to the parent process, which is then responsible for the writing when the worker objects are "finalized" (joined or terminated); this should reduce the per-task overhead, particularly in terms of IO - Now also constructed with `get_worker_finalization_patch()` line_profiler/_child_process_profiling/multiprocessing_patches /_queue.py::get_mp_pool_patch() Removed (superseded by `.._pool_patch_helpers.get_per_task_callback_patch()`) tests/test_child_procs/test_child_procs.py ::test_cache_setup_child(), _test_apply_mp_patches_inner() Updated debug-log messages to be matched, because the writing of profiling stats is no longer necessarily a result of a `Cleanup.cleanup()` call
(See `fable-pr431-plan-2026-07-05.md::P5` and `P6` in #5) kernprof.py::_manage_profiler .__enter__() Fixed bug where if the setups for the executed script and/or the session cache fails, the `CuratedProfilerContext` isn't cleaned up .__exit__() Fixed bug where if the gathering of child-process profiling stats fails, the process-local stats are not written
(See `fable-pr431-plan-2026-07-05.md::D.1`, `D.3`, `D.4`, `D.8–.10` in #5) tests/test_child_procs/_test_child_procs_utils.py FuncCallTimeout Renamed from `TestTimeout` so that `pytest` doesn't try to collect it as a unit-test class and emit a warning _run_kernprof_main_in_process() Now accepting `cmd` which begins with e.g. `['python', '-m', 'kernprof']` and other similar forms, instead of only `['kernprof']` (`D.1`) @cleanup_extra_pth_files - Refactored to only process .pth files which is consistent with the `LineProfilingCache.write_pth_hook()` naming scheme (`D.3`) - Added doctest @preserve_object_attrs Now emitting debug messages for teardown-time errors and re-raising the last (`D.9`) ResultMismatch.rich_message Removed unused property (`D.10`) CheckWarnings - Extended doctest to cover `.suppress_warnings()` and `.propagate_warnings()`, because it makes more sense to keep both methods for symmetry (`D.10`) - Removed implementations of `Sequence` mixin methods (the inherited impls are more inefficient, but we don't really use any of them in the code anyway) (`D.10`) tests/test_child_procs/conftest.py check_purelib_dir_writable() New fixture to be requested by tests, which tests for the writability of `sysconfig.get_path('purelib')` and skips the test if it isn't (`D.4`) another_pid() Replaced a (botched) subtract-and-mod with a bitwise XOR to ensure that the "PID" is both different and positive (`D.8`) tests/test_child_procs/test_child_procs.py test_running_multiproc_script() _test_profiling_multiproc_script() _test_profiling_bare_python() Replaced unqualified `kernprof` commands with `<sys.executable> -m kernprof` (`D.1`) test_cache_setup_main_process() test_apply_mp_patches_{success,failure}() test_profiling_multiproc_script_{success,failure}() test_profiling_bare_python_{success,failure}() Now requesting `check_purelib_dir_writable`, skipping the tests when the pure-lib path directory is non-writable (`D.4`) tests/test_child_procs/test_fork_stats_attribution.py - Added type annotations - Refactored to avoid awkward indentations (which trip up the syntax highlighter) - Tests now requesting `check_purelib_dir_writable`, skipping the tests when the pure-lib path directory is non-writable (`D.4`)
(See `fable-pr431-plan-2026-07-05.md::D.5-6` in #5) kernprof.py _write_preimports() Now returning `None` if the tempfile has been immediately used and deleted main(), _manage_profiler.__exit__() Added checks to file-cleanup/IO routines so that they are no-ops if we're in a forked child created inside the executed code; all such interactions with the "outside world" should be handled by the calling parent process tests/test_child_procs/_test_child_procs_utils.py preserve_targets .__doc__ Extended doctest to test `verify` .__init__() Added parameter `verify` .__enter__(), .__exit__() Updated implementation to store the dict of old values and compare them with the values at context-exit with `.compare_with_current_values()` if `.verify` is true _run_kernprof_main_in_process() - Updated decorator: `@preserve_targets(verify=True)` - Added check against `os.environ` pollution (`D.6`) tests/test_child_procs/test_child_procs.py _test_applymp_patches_inner() Updated decorator: `@preserve_targets(verify=True)` test_profiling_multiproc_script_{success,failure}() Reworked parametrization to: - Remove unintentional default (`start_method=None`) cases - Ensure that we run at least one subtest with `subprocess.run(['kernprof', ...])` for each combination of `start_method` and `fail` (`D.5`) test_profiling_bare_python_{success,failure}() - Updated call signature - Added subtests which test when the new Python process is created via `os.fork()` (`D.6`)
(See `fable-review-pr431-2026-07-05.md::D.6` in #5) tests/test_child_procs/_test_child_procs_utils.py ::check_tagged_line_nhits() Added new optional parameter `comparator` so that comparisons other than `operator.eq` can be done tests/test_child_procs/test_abnormal_execution.py New test module for various edge cases (`D.6`) test_prematurely_terminated_process() Test that if a process is `.terminate()`-ed, it impairs collecting profiling data therein but doesn't affect the rest of the profiling session test_corrupted_child_stats_file() Test that if a child-process profiling-data file is corrupted, the profiling data therein are lost, but that doesn't affect the rest of the profiling session test_unwritable_purelib_path() Test if we can't write a .pth file, while we cannot set up profiling in non-forked children, it doesn't affect the rest of the profiling session Note: we omit tackling nested `kernprof` sessions (`P13.8`), because there isn't a(n easy) way to make it consistent.
tests/test_child_procs/_test_child_procs_utils.py::concat_command_line()
Updated conditional check switching between the POSIX and Windows
implementations
tests/test_child_procs/test_abnormal_execution.py
<General>
Now printing the debug logs with `kernprof --debug-log=...`
test_unwritable_purelib_path()
- Now skipping the test on Windows since permissions are wonky
thereon
- Now directly using
`LineProfilingCache._enumerate_pth_installation_locations()`
to avoid missing possible paths
- Now also checking the group and other writable bits to ensure
that we can't write to the directories
TODO: see if this works on Linux...
line_profiler/_child_process_profiling/_retrieve_pids.py
::get_active_processes_windows()
- Removed erroneous check on the header line of the CSV data
(column-header strings can be quoted)
- Now wrapping errors raised during parsing in a `RuntimeError`
which shows the first lines in the `tasklist` output to help with
debugging
line_profiler/_child_process_profiling/_retrieve_pids.py
::get_active_processes_windows()
- Fixed bug where the parsing fails because the "PID" in the header
is capitalized
- Added stringification of the `.__cause__` to the raised error
because it is otherwise not visible (errors raised here are
converted to warnings)
(See `fable-review-pr431-2026-07-05.md::D.6` in #5) tests/test_child_procs/multiproc_examples/concurrent_test_module.py New test module with the same interface as the others (`pool_test_module.py`, `process_test_module.py`), except that `concurrent.futures.ProcessPoolExecutor` and `ThreadPoolExecutor` are used for parallelism tests/test_child_procs/conftest.py::concurrent_test_module[_object] New fixtures for the above test module tests/test_child_procs/test_child_procs.py ::test_{apply_mp_patches,profiling_multiproc_script}_{success,failure}() Updated parametrizations to use the new fixtures TODO: - New test for when a `BrokenExecutor` is raised; figure out how to trigger the error - Test fails when using a `ThreadPoolExecutor` with failing parallel workloads; figure out why
tests/test_child_procs/multiproc_examples/concurrent_test_module.py
Updated with a `gather_results()` similar to that of
`process_test_module.py`, so that instead of using `Executor.map()`
which errors out and may result in latter tasks not being run, we
always run each of the tasks to completion; this fixes the previous
"bug" where full profiling output cannot be extracted
line_profiler/_child_process_profiling/_patching_infrastructure.py
Migrated from `~.multiprocessing_patches._infrastructure` (WIP)
line_profiler/_child_process_profiling/multiprocessing_patches/
Updated import locations
line_profiler/_child_process_profiling/_patching_infrastructure.py
SingleModulePatch.submodule, .package
Removed
SingleModulePatch.module
Now the first field, superseding `.submodule`
Registry._default, .get_default()
Removed
line_profiler/_child_process_profiling/multiprocessing_patches/
__init__.py::get_registry()
Rehabiilitated from `Registry.get_default()`
__init__.py::_PATCHES
Removed (`get_registry()` is cached)
_pool_patch_helpers.py, *_patches.py
Updated `SingleModulePatch` instantiations
tests/test_child_procs/_test_child_procs_utils.py, test_child_procs.py
Replaced import of
`line_profiler._child_process_profiling.multiprocessing_patches._PATCHES`
with `.get_registry()`
line_profiler/_child_process_profiling/cache.py
::LineProfilingCache._get_graceful_online_call()
Fixed doctest which fails when run with vanilla `doctest`
_line_profiler_hooks.py::load_pth_hook()
If `line_profiler` can't be imported (e.g. we're in a
subinterpreter, which the Cython module is currently incompatible
with), we now return with a warning instead of erroring out
(See `fable-review-fullrepo-2026-07-05.md::F10` in #5) line_profiler/_child_process_profiling/multiprocessing_patches /_mandatory_patches.py Fixed bugs that resulted in child-process EoL (e.g. `atexit` hooks) being messed up and noisy error messages ("Exception ignored in `atexit`..." and "Exception ignored in weakref...") wrap_bootstrap() Now disabling the profiler at the end so that the `line_profiler._line_profiler` trace callbacks will be disabled before the interpreter is torn down, making sure that they won't be called when the underlying facilities (e.g. the `sys` module) are already starting to be GC-ed RESOURCE_TRACKER_PATCH New patch superseding `ResourceTrackerPatch`; instead of alerting the session to the resource-tracker server process and how it may result in an empty profiling-stats file, just clean up all the patches, tear down all profiling in the server process, and remove the profiling-stats tempfile it occupies; this avoids problems with resource lifetimes when said process outlives the profiling session in the main process, e.g. `LineProfilingCache.cache_dir` which is (normally) deleted by the main process at the end of `kernprof.main()` tests/test_child_procs/test_child_procs.py test_multiproc_script_sanity_check() test_running_multiproc_script() Fixed erroneous type hints for the `run_func` parameter test_profiling_multiproc_script_{success,failure}() - Fixed erroneous type hint for the `run_func` parameter - Now checking the `CompletedProcess.stderr` to make sure that the error messages are no longer emitted
tests/test_child_procs/_test_child_procs_utils.py
ModuleFixture
.uninstall()
New method for removing the module (if installed) from
`sys.modules` and the import system
.install()
Now returning a `Cleanup` which calls `.uninstall()`
._import_module_helper()
Simplified implementation
_run_as_{script,module,literal_code}()
Now `.uninstall()`-ing the installed `ModuleFixture`
tests/test_child_procs/test_child_procs.py
test_runpy_patches(), _test_profiling_bare_python()
Now `.uninstall()`-ing the installed `ModuleFixture`s
tests/test_child_procs/test_abnormal_execution.py
_revoke_write_access
- Refactored out from the body of
`test_unwritable_purelib_path()`
- Added class method `._check_usability()` to check whether the
context manager works as intended, optionally when a file in
the `chmod`-ed directory is open
- Now unsetting `stat.S_IWRITE` on Windows and `.S_IWUSR`,
`.S_IWGRP`, and `.S_IWOTH` otherwise
test_unwritable_purelib_path()
- No longer explicitly skipping on Windows
- Now checking if `_revoke_write_access` actually works and
skipping the `cannot-write-pth` subtests if not
tests/test_child_procs/test_abnormal_execution.py
::_revoke_write_access
- New (class-level-)cached property `.pid` wraps around
`os.getuid()` where available (i.e. not on Windows), and
`os.stat()` on an owned file otherwise
- Updated `.__enter__()` to use `self.pid` instead of the
platform-dependent `os.getuid()`
tests/test_child_procs/_test_child_procs_utils.py::VenvFixture
New helper class for creating temporary venvs and running Python in
them
tests/test_child_procs/test_child_procs.py
::_test_apply_mp_patches_inner()
- Fixed malformed regex pattern used to check for the logging MP
patch
- Now explicitly invoking `multiprocessing.util.debug()` to ensure
that the logging function has been reliably called
tests/test_child_procs/test_abnormal_execution.py
venv
New module-scoped `VenvFixture` fixture with `line_profiler`
installed from source
(note: test runs slower because of that)
test_unwritable_purelib_path()
Now using `venv` so that we always test running `kernprof` in a
venv; otherwise the disabling of writes to the purelib dir
doesn't necessarily work
line_profiler/_child_process_profiling/_cache_logging.py
::CacheLoggingEntry
Made parsing more resilient to malformed entries:
.from_text()
Refactored internals into separate static/class methods
._gen_timestamps_from_text()
- Helper method extracted from `.from_text()`
- Removed redundant capturing group
._gen_message_blocks_from_text()
Helper method extracted from `.from_text()`
._gen_entries_from_text()
- Helper method extracted from `.from_text()`
- Added doctest for failing cases: only the corrupted entry
should be affected, others should be preserved
- Added handling for various cases of malformed log entries:
- Replaced `textwrap.dedent()` call with `._dedent_block()`,
so that trailing text (e.g. when the next entry has a
malformed timestamp) doesn't cause the entire dedentation to
fail; a warning is then issued about the trailing text,
before the parsing moves onto the next entry marked by a
complete timestamp
- If the message header (containing the `.current_pid`,
`.main_pid`, and `.cache_id`) cannot be parsed following the
timestamp, a warning is issued about the malformed text
block and parsing moves onto the next block
._dedent_block()
New helper method for dedenting a text block which may be
"contaminated" by trailing text
tests/test_child_procs/_test_child_procs_utils.py
::VenvFixture._find_executables()
Now checking if `scheme='venv'` is available (e.g. Python 3.11+;
Python 3.10 w/Homebrew patch) before calling `sysconfig.get_path()`
with the argument, and falling back to the default if not
2 participants
This PR adds support for
kernprofto profile code execution in child Python processes, building on ongoing work (see Credits).Usage
The EXPERIMENTAL new flags
--no-prof-child-procsand--prof-child-procs[=...]are added to kernprof. By setting--prof-child-procsto true, child Python processes created by the profiled process are also profiled:1Note how the
sum_worker()calls are profiled:Highlights
os.system()andsubprocess.run()multiprocessing2concurrent.futuresmultiprocessing"start methods" ('fork','forkserver', and'spawn') tested to be compatible, where available on the platform--preimportsor via test-code rewriting) replicated in child processesExplanation
line_profiler._child_process_profiling.cache.LineProfilingCache) is created by the main process, containing session config information (e.g. values for--prod-modand--preimports) so that profiling can be replicated in child processes..pthfile is created; Python processes inheriting the right environment will thus go through profiling setup, while those without the env var (and just happens to share the Python executable) will be minimally affected.os.fork()(where available) is patched with a wrapper which ensures consistent global states.coverage.multiproc, variousmultiprocessingcomponents are patched (line_profiler._child_process_profiling.multiprocessing_patches.apply()) so that child processes can retrieve the cache and report profiling data appropriately. Patches are inherited by forked child processes and reapplied by spawned ones. Extra care is taken in ensuring that profiling is not affected even when the parallel workload errors out.3kernprofthen gather and merge with the profiling result in main process.Code changes
New code (click to expand)
_line_profiler_hook.pyNew module installed along with
line_profilerfor managing the temporary.pthfiles used for setting up profiling in child processes; this module is kept as lightweight as possible to minimize the amount of startup code run as the mere result of having said.pthfile(s) (presumably) in the virtual environment'ssite-packages.load_pth_hook():For processes inheriting a matching "parent PID" from the environment (see
LineProfilingCachebelow), load the cache and set up theLineProfilerinstance used, like how the mainkernprofprocess does.line_profiler/_threading_patches.pyNew submodule patching
threadingfor the consistent gathering of profiling data between tracing modes.apply():When legacy tracing is used (Python < 3.12 or
LINE_PROFILER_CORE=ctrace), patchthreading.Thread.__init__()so that the profiler's.enable_countis synced to the new thread; this is necessary for correctness, ensuring that profiling continues on the new thread.line_profiler/cleanup.pyNew submodule defining the
Cleanupclass, which handles various setup/cleanup tasks like:line_profiler/curated_profiling.pyNew submodule containing mostly relocated code from
kernprof, so that child processes can more easily reestablish profiling:ClassifiedPreimportTargets:Object resolving and classifying the
--prof-mods, and writing a corresponding preimport moduleCuratedProfilerContext:Context manager managing the state of the
LineProfiler, e.g.:line_profiler.profileon startupthreading(see_threading_patchesabove) so that the profiler stays enabled on newly spawned threads.enable_counts on teardownline_profiler/_child_process_profiling/New private subpackage for maintaining the states, setting up the hooks, and performing the patches which makes it possible to profile child processes:
/cache.py::LineProfilingCache:"Session state" object. It:
Can be auto-(de-)serialized in the main and child processes based on env-var values, managing setup (module patches,
.pthtempfile creation, profiler curation, eager pre-imports) and cleanup (tempfile management, dumping and gathering of profiling results) in each process.Injects the following environment variables, which are inherited by child processes:
${LINE_PROFILER_PROFILE_CHILD_PROCESSES_CACHE_PID}: main-process PID${LINE_PROFILER_PROFILE_CHILD_PROCESSES_CACHE_DIR_<PID>}: location of the cache directoryFrom the combination of both, child processes can retrieve the cache by calling
.load()./multiprocessing_patches/:Sub-sub-package for the patching of
multiprocessingfacilities so that child processes managed thereby are properly profiled. The main entities are:/__init__.py::apply():Select and apply the appropriate patches below to
multiprocessingmodule components./_mandatory_patches.py::PROCESS_SETUP_PATCH:Patch: perform setups specific to
multiprocessing-managed child processes./_mandatory_patches.py::POOL_WORKER_PID_PATCH:Patch: keep track of which
multiprocessing.pooltasks are sent to whichPool-worker child processes, so that we don't expect profiling output from idle workers./_mandatory_patches.py::RESOURCE_TRACKER_PATCH:Patch: make sure that profiling isn't set up on the resource-tracker server process, and if it is, tear it down. This ensures that when the profiling session ends and cleans up in the parent process, it wouldn't cause problems in the server process.
/_mandatory_patches.py::RebootForkserverPatch:Patch: manage the fork-server process (from which child processes are, well, forked; see
multiprocessing.forkserver), ensuring that (1) when profiling, it is rebooted with the proper profiling tooling which its forked children can inherit, and (2) when profiling ends, it is rebooted so that said tooling doesn't leak into future child processes./_mandatory_patches.py::RunpyPatch:Patch the copy of
runpythatmultiprocessing.spawnuses; necessary for profiling to function in non-eager-preimports mode (--no-preimports)./_profiling_patches.py::POOL_PATCH:Patch
multiprocessing.poolso thatPool-worker child processes dump profiling data after every task received,3 regardless of whether the parallel workload succeeded or errored out./_profiling_patches.py::PROCESS_PATCH:Patch
multiprocessing.process.BaseProcessso that child processes dump profiling data after the parallel workload (Process(target=...)) is executed./_optional_patches.py::LOGGING_PATCH:Patch the various logging functions in
multiprocessing.util(e.g.multiprocessing.util.info()) so that their log entries are also visible in theLineProfilingCachedebug logs./runpy_patches.py::create_runpy_wrapper():Make a clone of the
runpymodule which checks if the code executed is the code to be profiled; if so, it goes through the same code-rewriting facilities thatline_profiler.autoprofile.autoprofile.run()uses to set up profiling. (SeeRunpyPatchabove.)tests/test_child_procs/Refactored and greatly extended into a package from the previous
tests/test_child_procs.py; the tests proper are now found attests/test_child_procs/test_child_procs.py, while the example scripts/modules to be profiled have been moved to their own respective files intests/test_child_procs/multiproc_examples/./_test_child_procs_utils.py:Define various utilities, e.g.:
ModuleFixture:Helper object to be created as
pytestfixtures which encapsulates the/multiproc_examples/scripts, making them temporarily available as modules in the current and/or child processes.Params:Helper object extending
@pytest.mark.parametrize()which handles concatenation and Cartesian products of parametrizations.CheckWarnings:Helper object analogous to
@pytest.warns()allowing for easy checks for/against certain warnings issued.@add_timeout:Decorator for isolating a function in a new thread so that it can be timed out.
run_subproc():Wrapper around
subprocess.run()which provide extra debugging output (standard streams, timing info, etc.)/multiproc_examples/:The example scripts to be profiled.
/pool_test_module.py,/external_module.py:The existing example script (and the module it imports), where
multiprocessing.pool.Poolis used for parallelism./process_test_module.py:New example script mirroring
/pool_test_module.pywhich instead directly manages instances ofmultiprocessing.process.BaseProcessand communication of parallel workloads and results therewith./concurrent_test_module.py:New example script mirroring
/pool_test_module.pywhich uses aconcurrent.futures.Executorinstead of amultiprocessingpool, and deals with futures instead of tasks./conftest.py:Set up
_test_child_procs_utils.py::ModuleFixturefixtures for the/multiproc_examples/scripts so that they can be imported as modules in tests./test_pool_protocol.py:(Contributed by @Erotemic.) New tests making sure the profiling session is resilient to failures in setting up profiling in child processes; profiling stats therefrom may be lost, but it doesn't cause the executed code to fail or other stats to be lost.
/test_fork_stats_attribute.py:(Contributed by @Erotemic.) New tests making sure when child processes are created with
os.fork(), preexisting profiling stats already accumulated on the sessionLineProfilerobject aren't double-counted./test_child_procs.py:line_profiler._child_processing_profilingcomponents, or as close as is possible thereto:test_runpy_patches():Test the functionality of
~.runpy_patches.create_runpy_wrapper().test_cache_dump_load():Test the functionalities of
~.cache.LineProfilingCache.dump()and.load().test_cache_setup_main_process():Test the functionality of
~.cache.LineProfilingCache._setup_in_main_process().test_cache_setup_child():Test the functionality of
~.cache.LineProfilingCache._setup_in_child_process().test_load_pth_hook():Test the functionality of
~.pth_hook.load_pth_hook().test_apply_mp_patches_{success,failure}():Test the functionality of
~.multiprocessing_patches.apply().test_profiling_multiproc_script_{success,failure}():"Main" new tests for running the test scripts with
kernprof --prof-child-procs; heavily parametrized to check for profiling-result correctness in different contexts:success|failure: whether the parallel workload errors outrun_func: execution modes (kernprof <script>,kernprof -m <module>, andkernprof -c <code>)test_module: whether to runpool_test_module.py(testingmultiprocessing.pool) orprocess_test_module.py(testingmultiprocessing.process)prof_child_procs: whether to use child-process profiling (--[no-]prof-child-procs)preimports: eager vs. on-import profiling (--[no-]preimports)use_local_func: whether the parallel workload is locally defined in the executed code or imported from external modulesstart_method:multiprocessing"start methods" ('fork','forkserver', and'spawn')test_profiling_bare_python():New test for profiling child processes where the code run by
kernprof --prof-child-procsspins up another Python process via non-multiprocessingmeans (e.g.os.system()orsubprocess.run())./test_abnormal_execution.py:New tests for various edge cases where profiling is hindered on child processes for various reasons, testing that overall code execution and stats-gathering from other processes (incl. the main one) are not affected.
test_prematurely_terminated_process(): when themultiprocessing.process.Processis.terminated()and thus cannot properly clean uptest_corrupted_child_stats_file(): when theLineStatsfile written is corrupted and thus cannot be readtest_unwritable_purelib_path(): when a.pthfile cannot be written, and thus profiling cannot be set up on non-forked child processesModified code (fixes; click to expand)
pyproject.toml::[tool.ty.terminal]Now explicitly setting
error-on-warningto false because the default behavior changed inty v0.0.52. (Note that this conflicts with identical change made in #434.)line_profiler/line_profiler.py::LineStatsFixed doctest in multiple methods (
.__eq__(),.__add__(),.__iadd__(),.from_stats_objects()) which may give the wrong impression of the layout of the.timings; the dict keys are supposed to be of the layout(filename: str, lineno: int, func_name: str), not(func_name, lineno, filename).line_profiler/toml_config.py::ConfigSource.get_subconfig()Fixed bug where the subtable is not deep-copied even with
copy=True.kernprof.py::_prepare_profiler()Fixed bug in the pre-refactor
_pre_profile()wheresys.argvis replaced with another list, preventing the@_restore.sequence(sys.argv)decorator from correctly restoring thesys.argventries; also see the section below.Modified code (others; click to expand)
.github/workflows/tests.ymlAdded timeouts to job stages where
pytestis invoked, so that ifmultiprocessingcauses any of the new tests to deadlock the whole pipeline doesn't get stuck in limbo for hours.Build sdist -> Test full loose dist: 10 minutes<OS>, arch=<ARCH> -> Build binary wheels: 60 minutes<PYTHON_VERS> on <OS>, arch=<ARCH> with <EXTRA> -> Test wheel <EXTRA>: 10 minutessetup.pyAdded
_line_profiler_hooksto the installed among thesetuptools.setup(py_modules=[...]).line_profiler/line_profiler.py::LineStats.get_empty_instance():New convenience class method for creating an instance with no profiling data and the platform-appropriate
.unit..from_files():Added new parameters
on_defective | on_empty: Literal['ignore', 'warn', 'error'], allowing for passing over bad (empty/malformed) files with optional warnings. The old behavior (on_defective=on_empty='error') remains the default.line_profiler/line_profiler_utils.pyAdded new utilties:
CallbackRepr:reprlib.Reprsubclass helpful for formatting calls (via its.format_call()method) and adjacent objects (e.g.functools.partial).block_indent():Block-indent a multi-line string to sit flush with a prefix.
make_tempfile():Convenience wrapper around
tempfile.mkstemp()which returns apathlib.Path.line_profiler/rc/line_profiler.toml[tool.line_profiler.kernprof]:New key-value pair
prof-child-procs = <bool>for the default of thekernprof --[no-]prof-child-procsflag.[tool.line_profiler.child_processes]:New key-value pairs for controlling the setup of profiling in child processes:
pth_files = { prefix = <str>, suffix = <str>}:Instructions on how the temporary .pth files responsible for setting up shop in child processes are created.
multiprocessing = { patches = { pool = <bool>, process = <bool>, logging = <bool>} }:Toggles for which of the
multiprocessingpatches to apply.The
child_processestable and its contents are as of yet considered private implementation details.kernprof.py_add_core_parser_arguments():Now adding these flags to the parser:
--prof-child-procs[=...],--no-prof-child-procs:Whether to use the feature implemented in this PR; the default is read from
[tool.line_profiler.kernprof]::prof-child-procs.--debug-log=...:Undocumented (private) flag for writing out the
LineProfilingCachedebug logs._write_preimports():Refactored to use the new/relocated facilities at
line_profiler.curated_profiling._dump_filtered_stats():extra_line_stats: LineStats | Noneallows for handling and combining the profiling stats gathered elsewhere (e.g. child processes)._dump_filtered_line_stats()which it now calls._manage_profiler:Context manager refactored from the old
_pre_profile()for more Pythonic handling of setups and teardowns._prepare_child_profiling_cache()._prepare_profiler(),_prepare_exec_script())._post_profile()on context exit so that we no longer have to explicitlytry: ... finally: ...in_main_profile()._post_profile():extra_line_stats: LineStats | Noneallows for handling and combining the profiling stats gathered elsewhere (e.g. child processes).line_profiler.curated_profiling.tests/test_child_procs/test_child_procs.pyMass-relocated code which isn't directly the test functions into other locations in the package (see New Code above). Plus the following changes (among others):
/pool_test_module.py:Added the following command-line flags:
--start-methodselects a specificmultiprocessing"start method", includingdummywhich usesmultiprocessing.dummy.--localtoggles between using a sum function defined locally in/pool_test_module.pyor the one defined externally in/external_module.py.--force-failuretoggles whether the sum function should return normally or raise an error./conftest.py::pool_test_module:Supersede the previous
test_module./_test_child_procs_utils.py::ModuleFixture, allowing for easy setup as a temporarily import-able module in the current and/or child processes (see above).pool_test_module_clone(same source code but separate module) andprocess_test_module(same CLI but different model of parallelism)/_test_child_procs_utils.py::_run_as_{script,module}():_run_as_literal_code()to also testkernprof -c ....test_moduleas aModuleFixtureinstead of a path, and handling its installation./_test_child_procs_utils.py::_run_test_module():run_module = partial(_run_test_module, _run_as_module), etc. now available for more convenient testing ofkernprofexecution modes as test parametrization.profiled_code_is_tempfile: boolhelps with constructing thekernprofcommand line in cases where the code is anonymous (kernprof -c ...).use_local_func: bool,fail: bool, andstart_method: Literal['fork', 'forkserver', 'spawn'] | Noneallows for fuzzing code execution with the aforementionedtest_moduleCLI flags (resp.--local,--force-failure, and--start-method).nhits: dict[str, int] | None, when provided, checks that the line-hit stats are as expected (all calls traced with--prof-child-procs, only those in the main process without).subproc: boolfor toggling whether to run the test module in-process or in a subprocess.failis true, thekernprofsubprocess should fail..pthfiles created bykernprof --prof-child-procsshould be cleaned up.nhits(where available).subprocis false, that certain warnings aren't issued (e.g.LineStatswarning that we're trying to read profiling stats from an empty file), paralleling the checks in/test_child_procs.py::test_apply_mp_patches_{success,failure}().LineProfilingCachedebug logs and printing them for debug purposes./test_child_procs.py::test_multiproc_script_sanity_check():/process_test_module.py.use_local_func,fail, andstart_method, to ensure that the test scripts are fully functional in vanilla Python.as_module: boolwithrun_func: Callable[..., CompletedProcess], allowing for more flexible testing of execution modes (python ...,python -m ..., and the newpython -cadded via the aforementioned_run_as_literal_code())./test_child_procs.py::test_running_multiproc_script():New parametrization
run_funcallows for absorbing the oldtest_running_multiproc_module()into the same test as additional parametrization, as well as testingkernprof -c.Caveats
.pthfile created is course benign and as mentioned tries to be as out of the way as possible, but I just figured that the use of.pthfiles should be called out, given their recent spotlight in a CVE vulnerability..pthfile is written to a predetermined list of candidate locations (e.g.sys.get_path('purelib')), it depends on one of those directories being writable. If we aren't in a venv or a similarly isolated environment (which is increasingly unlikely nowadays), all processes using the same Python will have to import and run_line_profiler_hooks.load_pth_hook(). While the function itself should quit rather quickly when we're not in a child process, and without causing any ofline_profilerto be loaded intosys.modules, it does still incur a certain overhead for interpreter boot-up.multiprocessing.process.BaseProcess.terminate()andsubprocess.Popen.kill()) circumvents the Python interpreter, and thus disrupts cleanup and can result in profiling-data loss/corruption.initializerargument tomultiprocessing.pool.Poolandconcurrent.futures.Executorcauses a failure.What didn't work
multiprocessing-managed child processes have the usualatexithook which dumps profiling data disabled. On top of that themultiprocessingpatch "pool" (~._child_process_profiling.multiprocessing_patches._profilling_patches.py::POOL_PATCH) also disables the dumping of profiling data at the end ofmultiprocessing.process.BaseProcess._bootstrap()patched in by the patch "process" ([...]::PROCESS_PATCH). This was the conclusion after much head-scratching, where processes terminated (presumably) in the middle of aLineStats.write()call results in corrupted/incomplete data.SIGTERMthatBaseProcess.terminate()sends to the child and only dumped the stats then and there (along with normal end-of-callback stats-dumping), which sounds much more efficient than dumping stats for every task submitted to aPool. However:SIGTERMon Windows is generally noted to be inconsistent. See e.g. thiscoverage.control.Coverage._init_for_start()comment and this SO discussion that it refers to.multiprocessing-managed children is known to be messy and causes occasional deadlocks; a lesson I found out way too late and after much wailing and gnashing of teeth. See e.g. Deadlock in multiprocessing.pool.Pool on terminate python/cpython#73945, Setting a signal handler gets multiprocessing.Pool stuck python/cpython#82408, and Process hang with Coverage 6.3 coveragepy/coveragepy#1310. Guess that's whycoveragemade theirSIGTERMhandler opt-in after originally having that on by default.multiprocessing.poolis so gung-ho in killing off worker processes and replacing them, without much regard of what actually went down down there.3TODO
line_profiler._child_process_profilingto become public API?Credits
pytest-autoprofile, which in turn was based on the solution implemented incoverage.multiproc. And the longer I've worked on this, the more the similarities seem to have diminished ...LineStatsaggregation #380kernprofto always run code insys.modules['__main__']'s namespace #423Notes
Welp. This took way longer than I expected. (EDIT: I think I wrote this sentence like two months ago.) The main friction points were that:
line_profiler._child_process_profiling.cache.LineProfilingCacheclass tackles this issue.pytest-autoprofile::tests/test_subprocess.py::_test_inner()), but apparently I only made the tests fail there, not the parallel functions themselves. Figuring out how to do so consistently took the better part of these months.Footnotes
Note however that the equivalent vanilla Python command (
python -c ...) would error out, because functions sent tomultiprocessingmust be pickle-able and thus must reside in a physical file. This is sidestepped bykernprof's always writing code received bykernprof -c ...and... | kernprof -to a tempfile (ENH: auto-profilestdinor literal snippets #338). ↩In the test suite, process creation is tested both with the most common
multiprocessing[.get_context(...)].Pooland with self-managedProcessobjects. Different patches tomultiprocessingare responsible for profiling-data collection between the two situations, and they have been tested to work both (1) individually on their respective use-cases and (2) together without stepping on one another. Seetests/test_child_procs/test_child_procs.py::test_profiling_multiproc_script_{success,failure}(). ↩Since
multiprocessing.pool.Pool-managed child processes ("workers") are regularlyand wantonlyterminated, which bypasses Python control flow and prevents e.g.atexithooks from executing, we've taken to report profiling data fromPoolworkers on a per-task basis. ↩ ↩2 ↩3