feat: serve schemas based on which tags exist on gh

[jo-mueller]

PR following #564 that solves the problem a bit more elegantly (I hope):

This PR changes the build process of the page so that the following happens on build:

• All tags on ome/ngff-spec are automatically retrieved with git ls-remote --tags https://github.com/ome/ngff-spec
• This gives a list of tagged versions on that repo (currently the ones listed below)
• Copy all the schemas over here and serve them under the respective tag as .schema as well as .schema.json
• The schema publishing for the older versions (0.1...0.5 + dev) remains active as we still have the submodules in place here, from which we copy the schemas and serve them as well. Currently, schemas from the submodules supersede the schemas from the tags, but that's not a problem since none of the submodule'd schemas are from stagged version (i.e., 0.1...0.5 have not been tagged at ngff-spec)

This way, we are able to access the schemas of all past (we'd need to tag them) and present through the ngff page, including the dev versions. This also centralizes the tagging over at ngff-spec as a core mechanic to make schemas available through the ngff page.

To figure out next: If the 0.5 schema has changed, should it be published under 0.5 or under 0.5.X? which is a question we don't yet have answered.

cc @jni

[github-actions]

Automated Review URLs

• Readthedocs

[jo-mueller]

@lubianat do you have ideas how to make this nicer? I tried a few things with redirects, but maybe the RTD redirects supersede any redirects to outside pages (i.e. via an htaccess file) I am defining here, which is annoying.

To have the browser render the schemas as text instead of downloading the files instantly, we would need to rename the files into .json, I think.

[lubianat]

@jo-mueller should we go ahead and add the .json for the next releases (per #462)?

I don't have a suggestion on how to make it nicer, though

[jo-mueller]

We could serve them as both *.schema and *.json? For the time being, this could also happen here until we make the change at ngff-spec?

[lubianat]

We could serve them as both *.schema and *.json? For the time being, this could also happen here until we make the change at ngff-spec?

I don't have a good perspective on the implications. I guess we can do it at least transitionally, while tools move towards .json or .schema.json

[jni]

including the dev versions

I expect that we would want those to be hidden under a toggle/details/whatever.

[jni]

How do I review this? I don't know what you mean by "nicer", and looking at the preview build I only see 0.6rc0, no dev versions...

[jo-mueller]

@jni I think there's two aspects to this:

1. displaying the spec texts of all tagged versions
2. delivering the schemas for all tagged versions

This PR only addresses the second one. I hadn't thought about doing the same with all the tagged versions, but I think that'll be a bit more complex (but not much). I guess the approach to automatically getting all the tagged spec versions here, would be something like:

• remove the submodules from this repo entirely
• on build, get all the tagged versions (as this PR does it)...
• ....and clone all of them (which this PR doesn't yet do).
• The toctree that organizes the content of the left sidebar on https://ngff.openmicroscopy.org/specifications/index.html would then need to be built automatically as part of that CI.

On a plus side, that would get rid of all submodule shenanigans here, which I think would be a big plus ^^" On the downside, that would potentially be quite a bit of cloning/updating stuff as part of a normal build command?

[jo-mueller]

decision point: We serve the schemas as both .schema and .schema.json, in the future (on the 0.9dev branch) we rename the schemas to .json or .schema.json

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ngff-weekly-dev-update-thread/110810/140

[jni]

@jo-mueller So, on the schemas side: where are the schemas actually served from this PR? Can I get a RTD build URL? Also, can you re-trigger build maybe with an empty commit so we can see the 0.9.dev1 schemas get served?

I finally grokked the .schema vs .schema.json issue. I suggest serving both for now. 0.9.dev1 links to the historical .schema files, so let's punt on fully renaming to at least .dev2.

[jni]

Ok, found it:

https://ngff--581.org.readthedocs.build/0.6.dev1/schemas/label.schema

[jni]

Since currently the 0.9.dev1 schemas are not being published, I consider this a strict improvement and would advocate for it getting merged ASAP.

[jo-mueller]

@jni small thing that was missing here: I modified the code a bit so that schemas are copied to the _html_extra directory for

• all tagged versions of the ngff-spec repo (0.6.dev1...0.9.dev1)
• all submoduled versions (0.1...0.5)

Without this change, we would stop serving schemas for 0.1...0.5 :)

Also to keep in mind that once ome/ngff-spec#187 is resolved, we can do a bit less special-casing here and split the build process in two functions that

• function 1: clone/fetch all the schemas according to the tag
• function 2: Build and render the desired example documents for a subset of versions we want to display (0.1...0.6 + dev)

Edit: Also, the failing lint check points to some broken cross-links in the 0.9.dev1

For the Links > Internal check, the following failures were found:

• At ./_build/html/_temp_ngff_spec/index.html:626:

  internally linking to #spec:example:coordinate_transformation; the file exists, but the hash 'spec:example:coordinate_transformation' does not

• At ./_build/html/_temp_ngff_spec/index.html:655:

  internally linking to #orientation-values; the file exists, but the hash 'orientation-values' does not

[jo-mueller]

@jni ok, all the schemas are available on the RTD build:

• https://ngff--581.org.readthedocs.build/0.1/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.2/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.3/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.4/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.5/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6.dev1/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6.dev2/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6.dev3/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6.dev4/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.6rc0/schemas/image.schema
• https://ngff--581.org.readthedocs.build/0.9.dev1/schemas/image.schema

Edit And as a little extra: All the schemas are now also available with a .schema.json extension (e.g., https://ngff--581.org.readthedocs.build/0.9.dev1/schemas/image.schema.json) so the browser shows you the schema instead of downloading it.

[lubianat]

LGTM; I think it closes #462 too (maybe that in the commit message)

[will-moore]

@jo-mueller I'm trying to update the validator to dynamically parse each schema to determine what $refs it contains and then to load those schemas, instead of the previous hard-coded behaviour, e.g. for v0.6:

    const names = ["coordinate_transformations", "coordinate_systems", "axes", "_version"];
    for(const name of names) {
      const schema = await getSchema(getSchemaUrl(name, version));
      schema["$id"] = `https://ngff.openmicroscopy.org/0.6rc0/schemas/${name}.schema`;
      refSchemas.push(schema);
    }

The new code is a work in progress (PR soon) but it's working for most $refs e.g. https://ngff--581.org.readthedocs.build/0.6rc0/schemas/image.schema.json contains "$ref": "https://ngff.openmicroscopy.org/0.6rc0/schemas/coordinate_systems.schema#/$defs/coordinateSystem" so then I load https://ngff--581.org.readthedocs.build/0.6rc0/schemas/coordinate_systems.schema.json

However, that coordinate_systems.schema contains "$ref": "axes.schema" which doesn't look like a valid $ref.
This needs some tricky hard-coding to work around that.
I see the same "$ref": "axes.schema" in older schemas too, e.g.
https://raw.githubusercontent.com/ome/ngff-spec/refs/heads/main/schemas/coordinate_systems.schema

Any chance that could be fixed to be consistent with the other refs?
Thanks

[will-moore]

validator PR to test this PR is at ome/ome-ngff-validator#86

[jo-mueller]

@will-moore

Any chance that could be fixed to be consistent with the other refs?

Sounds like a real schema bug so we definitely should fix this!