`filewriter`: deployment topology

[Tom-Willemsen]

We need to define how the filewriter and other cooperating processes will run.

• One-per-instrument: each FW process incrementally writing out the events from a single instrument
• Pooled: some number, perhaps dynamic, of filewriters which may pick up a writing job from any instrument. This is the architecture that ESS use.

Things to consider:

• Pooled architecture is a requirement if we might ever collect data faster than it can be written to disk. This seems unlikely at present.
• Consider interaction with filewriter: intermediate files & autosave files #96 ; this likely makes a pooled architecture more desirable
• Needs to work for HRPD-X/SANDALS2/WISH/VESUVIO etc before MNeuData is available. We do not need to solve the MNeuData case in this issue, but we should pick an architecture which makes it possible to extend in likely MNeuData directions later.
• Consider interaction with filewriter: file archiving #85

[Tom-Willemsen]

@Pasarus your input is welcome, especially r.e. making this extensible towards likely MNeuData approaches.

[Pasarus]

An architecture that takes configuration values in either a config file or set of environment variables, with a preference for the latter, would be ideal for deploying en-mass in a typical operator pattern, allowing trivial horizontal scaling based on usage.

One thing to discuss is the requirement for a message queue, and likely how/where that will be hosted (can be inside the MNeuData compute cluster). I foresee a requirement for the message queue, due to the likely desire to queue up requests/provide a pool of compute for filewriters, instead of having 1 filewriter waiting for each instrument and needing a heck of a lot of more hardware, a generic filewriter that can handle any instrument as needed makes the most sense to me.

Another technical caveat will be ensuring that this is not a windows only, un-containerisable piece of software, whilst ensuring that the software should be able to redo a job if it is interrupted part way through (hence the message queue), that will get picked up by another in the queue if the filewriter disappears for some reason.

[Tom-Willemsen]

One thing to discuss is the requirement for a message queue

If we choose a pooled architecture, then I think the ESS schemas & kafka topics for FW control would be the starting point for this discussion. Doesn't mean we can't diverge if we deem it necessary.

Another technical caveat will be ensuring that this is not a windows only, un-containerisable piece of software

It will run on Linux (likely in a container) for HRPDX/SANDALS/WISH/VESUVIO etc. As it just has Kafka as input and filesystem as output, I see no special reason why it would be particularly difficult to containerise.