Staring python on a HPC system with lustre can cause significant overhead. We should therefore investigate to avoid starting PSyclone as a subprocess, instead the PSyclone tool could just call psyclone.generator.main(...) directly.
Since PSyclone was not developed originally with this in mind, there is the risk that this change could cause issues (i.e. state information from processing one file could leak into another file). One example would be the module manager in PSyclone, which is a singleton. But this example should actually be advantageous, since the file information does not need to be recollected and can be reused, actually reducing file system accesses since the module manager caches a lot of information).
We had a ticket for this (stfc/PSyclone#838), but have closed this since at this stage it seems to work. If there are any issues, please re-open this ticket with details.
Re the implementation there are (at least ;) ) three options:
- We add a new PSyclone tool:
PSycloneSubprocess (the current PSyclone) and a new PSyclone tool (to me it seems sensible to have the no-subprocess to be the default??). If they are added to the ToolRepository in the preferred order, since by default the first (in this case PSyclone) tool will be used. Any site can overwrite this by picking one of the two PSyclone tools and adding it to the Toolbox.
- We completely remove the current tool (or change it to avoid a subprocess). Makes it all easier, but if there should be issues (including issues by other sites), this might be more complex to fix. Maybe a two phase approach? Deprecating subprocessed-PSyclone now, and removing it later?
- We leave one tool and support both operations. Each site can then set if the tool should use a subprocess or not (and we can manage the default in Fab).
TBH, I don't see much of a difference between these three approaches. The only minor thing that needs to be remembered is that we also need to change the way we get the PSyclone version number (which by default relies on running psyclone -f, while with the non-subprocess implementation it should just get the version number from psyclone.version.
Sam Clarke-Green (@t00sa) , I think you would be the person to look at this? Am happy to discuss.
Staring python on a HPC system with lustre can cause significant overhead. We should therefore investigate to avoid starting PSyclone as a subprocess, instead the PSyclone tool could just call
psyclone.generator.main(...)directly.Since PSyclone was not developed originally with this in mind, there is the risk that this change could cause issues (i.e. state information from processing one file could leak into another file). One example would be the module manager in PSyclone, which is a singleton. But this example should actually be advantageous, since the file information does not need to be recollected and can be reused, actually reducing file system accesses since the module manager caches a lot of information).
We had a ticket for this (stfc/PSyclone#838), but have closed this since at this stage it seems to work. If there are any issues, please re-open this ticket with details.
Re the implementation there are (at least ;) ) three options:
PSycloneSubprocess(the current PSyclone) and a new PSyclone tool (to me it seems sensible to have the no-subprocess to be the default??). If they are added to the ToolRepository in the preferred order, since by default the first (in this case PSyclone) tool will be used. Any site can overwrite this by picking one of the two PSyclone tools and adding it to the Toolbox.TBH, I don't see much of a difference between these three approaches. The only minor thing that needs to be remembered is that we also need to change the way we get the PSyclone version number (which by default relies on running
psyclone -f, while with the non-subprocess implementation it should just get the version number frompsyclone.version.Sam Clarke-Green (@t00sa) , I think you would be the person to look at this? Am happy to discuss.