calcYupYdown as needed?

[johnomotani]

In #1306 @d7919 commented

we're transforming both input fields and then transforming back because one or other don't have yup/ydown -- would we rather call calcYupYdown on one or more of the inputs instead? This would avoid having to duplicate code and should avoid the need to transform the result field so I think might be faster (assuming toFieldAligned costs about the same as a calcYup/down)?

toFieldAligned and fromFieldAligned cost 1 FFT/rFFT of a Field3D each. calcYupYdown is 2 but doesn't need the transform back so if one field had yup/ydown already it would be cheaper (2FFTs vs 3) to call calcYupYdown.

I think it would be a nice feature to have something smarter for when to calculate yup/ydown fields. Opening an issue to discuss since it's a longer-term issue than just a bugfix.

At the moment calcYupYdown() is called only as part of Mesh::communicate(). If we have 2 fields that have yup/ydown and then do some arithmetic with them (e.g. p=n*T), at the moment you have to either communicate p or do p.splitYupYdown();p.yup()=n.yup()*T.yup();p.ydown()=n.ydown()*T.ydown() by hand for every operation.

sub-issues:

• you might not want to call calcYupYdown() on a field if you aren't going to take a y-derivative. Actually this is already possible by using communicateXZ() instead of communicate(), but I don't think we've documented this feature in the manual.
• I don't think we document that it's possible to do things like p.splitYupYdown();p.yup()=n.yup()*T.yup();p.ydown()=n.ydown()*T.ydown() if you need them.
• It might be nice to include yup/ydown fields in arithmetic, e.g. if both arguments to an operator have yup/ydown fields, then calculate yup/ydown for the result as well. The downside for optimizing is that which fields are getting yup/ydown calculated is maybe too well hidden from the user; it would be rather obscure to deal with. Also, you might not want all arithmetic operations to do the same thing, e.g. I calculate both p and nu from n and T and I want p to have yup/ydown, but don't need them for nu.
• A nice compromise might be to call calcYupYdown 'on demand' as @d7919 suggested, then we definitely aren't carrying around yup/ydown fields when we don't need them, even if doing yup/ydown arithmetic might be more efficient sometimes. I think I'd be in favour of this even if it isn't actually faster: doing parallel derivatives with two different schemes makes me slightly uneasy (although I have no logical basis for it). I'd be happier if the code defaulted to doing either all yup/ydown derivatives or all to/fromFieldAligned derivatives.
  • could we have some kind of 'optimize warnings', maybe only active when CHECK>0? I'm thinking that if calcYupYdown() needs to be called in a derivative, it probably means that the user could speed things up by adding by hand calculations for yup/ydown, like that p.splitYupYdown();p.yup()=n.yup()*T.yup();p.ydown()=n.ydown()*T.ydown(). Then a user could write the simplest code to start with, and get hints for where to start optimizing without having to have a deep understanding of the ShiftedMetric and yup/ydown architecture already. In New shifted metric #1177 I added functionality for caching the field-aligned version of a field which has a similar effect to calling calcYupYdown on demand, since toFieldAligned() was already called whenever it was needed, I also added a message suggesting optimization might be possible (although a shinier system for collecting them and presenting them to the user would be nice), see

    BOUT-dev/src/mesh/parallel/shifttofieldaligned.cxx

    Line 261 in 55d87c8

    const Field3D ShiftToFieldAligned::toFieldAligned(const Field3D &f, const REGION region) {

[ZedThree]

I need to think about this more, but "optimisation warnings" could work really well. Maybe we can't actually do the fastest thing automagically, but we might be able to spit out a hint for improvement. This could be useful where something might need to be communicated as well.

[bendudson]

I think a while ago we discussed the possibility of printing messages when particular solvers are used, for example to print citations which should be used. If such a mechanism existed, perhaps it could be used to print this kind of optimisation hint too.

[johnomotani]

Fabio had an interesting suggestion: maybe it would be good to have some 'optimization' switches (either configure or runtime) to turn off expensive features that might not always be needed. Advanced users could call them explicitly where needed. calcYupDown might be a good example: we could have a switch to turn off the automatic calls (in Mesh::communicate and if they are added in derivatives, etc) and leave it up to the user to find the best places to put them to minimize the number of calls.

[ZedThree]

One possible way to Do the Right Thing is overload the y-derivatives on r-value references (i.e. temporaries, function return values) and then communicate/calcYupYdown. There's probably a whole bunch of different way this could go wrong, but could be useful.

It might also be possible to provide some function that takes two fields and applies another function to them and their yup/ydown. I don't know how to make this work yet, but again, could be useful.

---

On a side note, I've been thinking about names a bit recently. What nature of thing are yup/ydown? The ParallelTransform applies some nature of shifted metric -- but ShiftedMetric has already taken this name, even though FCI is a kind of shifted metric too.
yup/ydown are also some kind of magnetic field aligned fields, but then toFieldAligned does a different kind of thing again.

How about "parallel slices"? The yup/ydown fields are slices in the parallel direction. calcYupYdown might then become calcParallelSlices, Field3D::ynext becomes Field3D::parallelSlice

[johnomotani]

The simplest way to get calcYUpDown called on-demand seemed to me to be to make yup_field and ydown_field mutable. I think it looks OK, see #1342. But there might well be a better/nicer way...

I think the "parallel slices" naming would be good 👍

[ZedThree]

@johnomotani I have a WIP implementation that hopefully solves a bunch of problems. I'll open a PR for it so you can take a look and let me what I've missed!

[dschwoerer]

This seems to be outdated?

At least, I do not think it is relevant to the current, FCI capable, yup ydown handling anymore ...

@ZedThree @johnomotani @bendudson please reopen (open dedicated issues) if you think (some of) this still needs to be tracked ...