Move file creation to compile loop

[philipstarkey]

Original report (archived issue) by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

Currently if one clicks engage all h5 files get created and then send of to the compiler loop for processing of the labscript.

I think it would be better to send the compiler loop the parsed globals and let it create the h5 files individually when needed. This should reduce file clutter of aborted compilation. Only files of fully compiled shots would be in the output folder after pressing abort and not a mix of compiled an uncompiled shots. I'm guessing that this is what most users expect to happen anyway.

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

With this inplace we should also think about adding a context menu to the abort button where one can select the engaged sequence to abort.

In our lab we mostly want to abort the currently running compile process, but I could think of cases, where one would like to remove previously engaged sequences from the compile queue. This is a bit tricky as queues only support removing items from the front. But could be done with a set of sequences kept separately. The compile loop could check for membership of the currently compiling sequence in the set. If the sequence is not present (anymore) it would just stop compiling it and start the next.

[philipstarkey]

Original comment by David Meyer (Bitbucket: dihm, GitHub: dihm).

---

This would be great in my opinion. I have so many aborted h5 files lurking around.

Beyond that, could I also propose that the globals evaluation portion of globals parsing be moved into loop as well? We like to cycle our experiment while peaking up signals based on shot results. Right now the best we can do is repeat the same shot in BLACS and re-engage if we want to change any of the global parameters. Unless I'm much mistaken, moving the evaluation would mean we could implement our functionality with a simple while loop on the engage button.

Of course now that I've actually thought about this some I'm realizing that it would probably be better to use a flag to trigger a re-evaluation in loop since normal operation certainly doesn't need the same globals evaluated to the same values for each shot in the sequence.

In any case, however we implement our compile loop idea we would definitely need the file creation to be in loop so I'm on board with this idea, for what that's worth.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

In order to support per-shot feedback in the middle of a sequence (for example to keep magnetic field cancellation as accurate as possible based on a measurement each shot), I'm planning on introducing a compilation queue to runmanager, so that you can have the shots compile 'just in time' as BLACS needs them, evaluating the globals at the time of compilation.

As David points out though, the point of a queue is to be able to have elements in the queue be different! If you can't 'lock in' the values of some globals at submission time, then that would not be great. So I'll need to think of how you can specify either on a per-global basis or a per-click-of-the-engage-button basis, whether the globals are to be 'just in time' or 'as submitted'.

@PhyNerd, as mentioned elsewhere, the engage button is in the mainline runmanager greyed out when you have clicked it, so the distinction of 'abort one sequence' or 'abort multiple sequences' doesn't exist. If you guys have locally made changes that make the engage button remain active, then you may see undefined behaviour - I didn't intend for multiple compilations to be 'queued up'. However if I introduce a compilation queue, it could contain a treeview of sequences containing shots, and you could simply delete the ones that you want to abandon compilation for, either all of them, or on a per-shot or per-sequence basis. (and the shot files will not have been produced yet at that point).

The abort button could be relegated to aborting merely the current shot's compilation, which I could make it do more aggressively by terminating and restarting the compiler process, whereas at the moment it does not actually interrupt compilation mid-shot.

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

No it's not grayed out in mainline code I checked and wrote that over at #45. If it where grayed out the compile queue wouldn't make any sense that is in the mainline code as well.

And there is also a issue about the abort stuff from @shjohnst at #33 so I'm not the first to stumble upon this.

btw I fixed both #45 and #33 in a pull request I made yesterday.

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

I really like the treeview idea as it gives a lot more control to the user and is more visual than a context menu.

As for aborting the currently compiling shot I don't see much benefit in that other than if there is a endless loop or something like that going on.

[philipstarkey]

Original comment by Chris Billington (Bitbucket: cbillington, GitHub: chrisjbillington).

---

Ah, I see about the button not being greyed out. I just overlooked that - but I definitely didn't intend an actual queue. The fact that it's implemented as a queue object in Python is just because I like the actor model as a programming style as a result of playing spacechem (I am wearing a spacechem T-shirt right now). The fact that it nonetheless works is a testament to the actor model :p.

Well we'll make the queue more explicit, we'll just have to decide on a mechanism for marking which shots or globals are to be 'just in time' determined vs locked in at engage-time. And then we can integrate those progress bars you've made a pull request about.

About aborting the currently compiling shot, it wouldn't be worth putting much effort into, but since the functionality is already there in the form of the 'restart subprocess' button, it would be pretty simple to use it, but I won't bother for the moment if it turns out to be tricky.

[philipstarkey]

Original comment by Jan Werkmann (Bitbucket: PhyNerd, GitHub: PhyNerd).

---

Never heard of that game but it sure looks like fun.

Well as currently everything is locked at engage-time I would say that we start off with having everything determined at engage-time. This is a really quick thing to to by just adding sets of globals to the queue instead of files and compiling and creating files in the loop.

In the next step we could then think about a mechanism of marking shots to be evaluated at runtime and add that information to the sets of globals.

Those progress bars also have a bugfix that keeps the abort button from greying out with a non empty queue (issue #33). This is really needed for groups queueing up shots like we do as there is no way but to restart the subprocess to stop compilation atm.

[zakv]

This issue, at least as described in the title and initial post, seems to have been resolved. The runmanager.make_run_files() function returns an iterator which will generate files as they are requested, rather than generating them all right when make_run_files() is called. The generator then creates the files in the compile loop here.

It's also quick to test that this issue is resolved. Simply start compiling a bunch of shots and then abort the compilation and check what files exist. Only the files for the compiled shots will exist and no files will exist for the yet-to-be-compiled shots.

The discussion here touched on other issues as well, but maybe it would be best to close this issue and discuss the other issues elsewhere. Issue #33 covers the discussion about the abort button behavior, but it might be best to open a new issue to discuss using 'just in time' or 'as submitted' values for globals.