Speed up calculations of dependent parameters by bypassing unit handling

[henrikjacobsenfys]

General

Evaluating dependent parameters is slow. This means that especially computationally heavy things, such as Bayesian analysis on a global fit, can take hours. Initial tests indicate that it can be sped up to take minutes at most.

The primary reason for the slowness (i.e. more than 99% of the calculation time) is that we handle units every time a dependent parameter is called. We create a bunch of parameters, which create a bunch of scipp variables, we then convert the units etc. etc.
This is pointless in a fit: the output unit is the same throughout the fit, and it's a simple factor that can be multiplied in the end.

Idea

Add a dependency_compiler (name pending review) which, well, compiles the dependency expression and stores the final unit and any conversion factor needed to get the correct result. This is done only once during e.g. a fit. Then evaulate the expression using numpy functions (see #297 ) on the raw values of the parameters.

Disclosure:

I prototyped the code already with Claude. We get a speedup of about a factor 1000 for a typical EasyDynamics use case..
Edit: Ran some profiling, and those numbers are far less impressive - about 70% of the calculation time can be saved, so "only" a factor 3 improvement. Will update with more information when I have it.

[rozyczko]

This is a good idea but should be made an issue on the EasyCore board instead. This sounds like a bug and an idea for a bugfix...

[henrikjacobsenfys]

It's not really a bug, I think, since it does the right thing. It just does it sllloooooowwwwwlllyyyyy

[damskii9992]

Question: How many dependent parameters do you have in a typical fit Henrik? We originally only designed dependent parameters to be used for a few parameters . . .

[damskii9992]

But are we talking 16 Parameters here all-in-all or 16 Dependent Parameters with god knows how many other parameters behind the scenes?

[henrikjacobsenfys]

16 dependent parameters and another 16 independent ones in this case (would be 24, but we didn't simulate a background)

[henrikjacobsenfys]

In the more advanced scenario above it's probably typically 2-3 dependent and 2-3 independent parameters per Q. So with 16 Q that's 32-48 dependent and 32-48 independent parameters.

Of course I plan to also allow fitting simultaneously along other axes such as temperature, so we can multiply those numbers by a factor of 5-10 for the really fun cases 😀

[damskii9992]

16 dependent parameters should be within scope of how I imagined they would be used. The 30-50 possibly times 5-10 is DEFINITELY NOT! But if it is slow with 16 then we should definitely speed it up . . .
Is each dependent parameter only dependent on 1 independent parameter? Or how did you set up your dependencies?

Regarding your more crazy requirements, each Q and each temperature is an independent fit, right?
I will need to do parallel fits in easyimaging too, so this just sounds to me like we have two users of the potential parallel parameter implementation that I have planned for the future :)

[henrikjacobsenfys]

The area depends on a single parameter (and the expression is trivial, it's equal to that parameter), and the width depends on two parameters (Gamma=hbar D Q^2/(1+D tau Q^2)), D and tau are global.

The first step is always independent fits, and that will be enough for many people. But for the rest, when you're using a diffusion model (or another global model, they don't exist yet, but will), the whole point is to simultaneously fit all your data, or at least all Q. Typically with a local background, potentially other local model components and 1-2 components with dependent parameters that depend on a few global parameters.

[damskii9992]

Also, my proposed solution to #291 should give some speed-up.

[henrikjacobsenfys]

Perhaps in some cases, but not for the example that prompted this. The expression is already fully reduced.

[damskii9992]

Yes, but my proposed solution will not convert units that are already in the correct unit. So we will skip some unit-conversion and thereby Scipp scalar creation :)
It is only part of the performance that you complain about, but it is something.

[henrikjacobsenfys]

Where's the "clap hands above head, it's something" reaction? 😆

[henrikjacobsenfys]

Parsing the expression only once (i.e. during creation) and passing the parsed expression to asteval should also be a speedup of about a factor 3, I believe. Nothing compared to 1000, but still quite decent for very little work.
Edit: looks like my numbers were greatly exaggerated - see below

[damskii9992]

Could you please try and profile your fit Henrik?
I'd like to see a profile of where the fit spends most of the computation time.

[henrikjacobsenfys]

I'll give a more detailed reply later, but initial findings on a single fit make the total gain much less impressive than I quoted above. It'strue we can zpeed up dependent parameter calculations by A LOT, but we also do other things. I'll need to investigate more.
~70% of the time in a fit is spent doing uselss stuff as mentioned above, which can be almost entirely removed by optimising.
~30% of this time is spent in global object registering new parameters that get discarded again. The majority of that time is spent figuring out what the new parameter should be called.
5% is spent by asteval compiling the expression every iteration
~65% of the time is spent evaluating units

[henrikjacobsenfys]

This means that by far the simplest improvement is to make unique_names never repeat. That alone is a 30-40% speedup!

[damskii9992]

Could you please post the profiling here for all of us to see?

[henrikjacobsenfys]

Yes, but not when I'm focussed on keeping a child alive :) I want to give it proper attention and ensure it's done correctly. Tomorrow or Friday