xarray DataTree

[TomNicholas]

Hi there, I was just wondering if you had seen the prototype xarray DataTree we're working on? Supporting multi-scale image arrays is supposed to be one of the use cases so if you have any thoughts I would love to hear them!

[joshmoore]

@thewtex: writing here what I would have said last night on the zarr call had we not run out of time:

• had a good chat with Tom et al. re: Multiscale convention zarr-developers/zarr-specs#125
• he pointed out that dataclasses plays a related role to spatial-image
• and that assuming dataclasses can be mixed with datatrees(!) that the combination could play a similar role to spatial-image-multiscale
• (in a separate email, he mentioned xarray accessors but we didn't chat about that)
• all along way to say that you might be an ideal candidate (a) for testing out prototypes and (b) identifying functionality that you'd like to see added upstream 😉

[thewtex]

Hi @TomNicholas @joshmoore ,

Thank you for the tips!! 🙏

Yes, I will move forward with these excellent suggestions!

Current plans (thoughts welcome):

1. Investigate how we can create tests to validate adherence to the OME-NGFF spec
2. Update the SpatialImage type alias to a data class with further type specification
3. and that assuming dataclasses can be mixed with datatrees(!) that the combination could play a similar role to spatial-image-multiscale <- this =-)
4. add features to update output the OME-NGFF to the v0.4 spec

(in a separate email, he mentioned xarray accessors but we didn't chat about that)

It is later down the line, but I am interested in your thoughts on how we could possibly use this to make itk.Image more compatible with xarray.

[thewtex]

Investigate how we can create tests to validate adherence to the OME-NGFF spec

The ngff repository now makes JSON schema validation possible per:

• https://github.com/ome/ngff/blob/main/.github/workflows/validation.yml
• NGFF dataset validator ome/ngff#58

Future, additional array validation is being added per:

• Validate data ome/ome-zarr-py#102
• Initial validate support, with --strict option ome/ome-zarr-py#142

Update the SpatialImage type alias to a data class with further type specification

Draft: spatial-image/spatial-image#8

[thewtex]

Draft implementation based on xarray DataTree in #10

SpatialImage was rebuilt on xarray dataclasses in spatial-image/spatial-image#8 and the constructor was refactored to correspond to the NGFF v0.4 data model. Todo: Add support for additional dimension orders.

[thewtex]

Channel-first support added in spatial-image 0.2.0

[thewtex]

Full initial implementation in #11

To come: more examples

[thewtex]

NGFF 0.4 validation in #18

[thewtex]

Basic docs: in #20

[thewtex]

Rename + updates to xarray-datatree 0.0.5 refactors in #35

[TomNicholas]

Glad you were able to update to update to accommodate those breaking changes in datatree so quickly @thewtex - let me know if you have any particular issues or suggestions!

[thewtex]

@TomNicholas if you please review / merge / cut a release with xarray-contrib/datatree#90 that would be very helpful!

This is a blocker to create demos e.g. re #32 and related to https://github.com/thewtex/shardedstore . With these demos @joshmoore this is worthwhile discussing with the zarr community.

[thewtex]

@TomNicholas again for your help with this!

We moved to the recently migrated xarray package DataTree in multiscale-spatial-image 2.0.0: https://github.com/spatial-image/multiscale-spatial-image/releases