corrections around data checker GPU and transfer optimisation for save_intermediate

[dkazanc]

pls ignore the name for the branch :)

1. It removes the insertion of data_checker (GPU) after save_intermediate_data method. Data is being checked before save_intermediate_data. So the order is more logical now:

    total_variation_PD (httomolibgpu)
    --- output check with data_checker --- (httomolibgpu)
    save_intermediate_data (httomo)
    calculate_stats (httomo)

compared to:

    total_variation_PD (httomolibgpu)
    save_intermediate_data (httomo)
    --- output check with data_checker --- (httomolibgpu)
    calculate_stats (httomo)

2. More major change around the transfer in SaveIntermediateFilesWrapper. Now the block can be transferred wholly to CPU, only if the method that follows is a CPU method. That saves of transferring from Device to Host twice. If the method after is a GPU method or there is no method then we only copy block to CPU in order to be saved, while the block itself remains on the GPU.

To provide a bit more context on this change: previously, save_intermediate_data did not transfer blocks to the CPU, so the data was always left on the device. However, the calculate_stats method, which usually follows save_intermediate_data, can operate on CPU data. Since the calculation is very simple, there is no need to provide a GPU array as input for this method.

Another consideration was that the data was being transferred from the device to the CPU twice: once for save_intermediate_data and then again for rescale_to_int, which is a CPU method. This change removes the second transfer by converting the blocks to CPU data inside save_intermediate_data. The resulting CPU block can then be passed directly to calculate_stats, rescale_to_int, and save_images, which are all CPU methods.

However, we do not transfer the data to the CPU unconditionally, because the next method in the pipeline after save_intermediate_data may still be a GPU method—for example, denoising after reconstruction. In that case, the GPU block remains on the device. A parameter has been added to control this.

3. I also removed nan_to_num bit from calculate stats wrapper as all the data should be corrected with data checker beforehand.

One of the main reasons of doing this change is trying to understand the GPU transfers related numbers for the paper.
So to add more context, this was the result on the Blackwell node before the changes and this stuff around calculate_stats really confused me, like why is this the GPU kernel time, why it is so long as well.

And this after the changes I've made. calculate_stats is purely on the CPU as the block-data already converted beforehand in save_intermediate_data method. It is indeed faster (few seconds for this small data but when we scale up it will be more noticeable).

Checklist

• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have made corresponding changes to the documentation
• I have added the user-release-note label in order to include this PR in the "Notable
  Changes for Users" section in release notes

[yousefmoazzam]

I can follow changes 1. and 3. but I struggle to understand change 2.

Given the attached benchmark plots it seems like the PR does what's intended so I don't think there's any need to make further changes to the function of the code, but it'd be nice to have a bit more clarity in the PR text on how the optimisation is achieved.

For example, there's a mention of transferring from device to host twice prior to this PR and that after the PR this is no longer the case. But I don't really understand how this was happening before, or why it's no longer happening now. Having a brief description of what was happening before would be useful (I think there's already a description of what happens after?).

[dkazanc]

thanks @yousefmoazzam I just added a bit more info on the second point. Hope it make more sense. Cheers