Update post processing scripts and docs

[Andrew Coughtrie (andrewcoughtrie)]

Description

PR #254 added a proper header to the output files including a format version, this PR bring the post-processing in line with that as well as updating the documentation to describe the new tooling a bit more clearly.

Linked issues

Closes # (issue)

Type of change

• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How has this been tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• New tests have been added
• Tests have been modified to accommodate this change

Checklist:

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes, for both debug and optimised builds

[Ed Hone (EdHone)]

In principle I'm happy with all the changes and their efficacy is demonstrated by the successful CI run. My only concern is that this would make the release version of the library only work with vernier data created from this release or later. That is a valid approach, but I just wanted to confirm that's the plan.

Also, the file format version is 1.0, will this version number only change when the file format is changed, or will it match the version of the software?

[Andrew Coughtrie (andrewcoughtrie)]

In principle I'm happy with all the changes and their efficacy is demonstrated by the successful CI run. My only concern is that this would make the release version of the library only work with vernier data created from this release or later. That is a valid approach, but I just wanted to confirm that's the plan.

Also, the file format version is 1.0, will this version number only change when the file format is changed, or will it match the version of the software?

For the format version it is intended to only change if the format changes, it may remain at 1.0 for all eternity, it should only change if there is a change to the output format of vernier that breaks the post processing, that way there is a way to link the post-processing tooling to files which work, it would be unsustainable to try and maintain backward compatibility for all time.

In terms of the ability to use the post-processing scripts with older versions of the output, that's a tricky one, if we don't do so now we will never stop supporting the older format I think. At the moment people with older outputs can probably just add the header to the top of their files and it will likely work though it will not be guaranteed to be correct, if it doesn't however they may get a horrible error.

[Ed Hone (EdHone)]

Happy with that response - this has passed my code review