MM-OME-Zarr-Storage

A high-performance Java library for streaming multi-dimensional microscopy images to OME-Zarr (OME-NGFF v0.5 on Zarr v3), with a multiscale pyramid and concurrent read-while-write. It is modeled on NDTiffStorage — same axes-keyed putImage API, single writer thread with a bounded queue for back-pressure, and an in-memory write-pending buffer so images are readable the instant they are queued — but writes the community-standard OME-Zarr format instead of a custom TIFF format.

The library has no dependency on Micro-Manager. Its API is deliberately shaped so a Micro-Manager Storage adapter is trivial to write (see docs/micromanager-adapter.md and the runnable example under src/test/.../examples/).

Features

• OME-Zarr v0.5 / Zarr v3 output, readable by napari, Fiji, and the zarr-python v3 stack.
• Multiscale pyramid with two ways to produce lower-resolution levels:
  • automatic 2×2 block-averaging on putImage (configurable number of levels), and
  • explicit — write caller-supplied data to any level at any time via the resolutionLevel overload.
  • Pyramid depth can be increased at any time with setMaxResolutionLevel, which back-fills the new levels from existing data.
• Concurrent read-while-write: reads during acquisition are served from an in-memory write-pending buffer, then from disk; chunk and metadata writes are atomic (temp file + rename) so out-of-process readers never see a torn chunk.
• OME-NGFF metadata (multiscales, axes, omero) plus a reserved micro-manager namespace for opaque summary and arbitrary custom metadata. Per-image metadata is kept in an append-only NDJSON sidecar.
• Pluggable storage engine via the ArrayBackend SPI; the default is a pure-Java zarr-java backend. A native/tensorstore backend could be dropped in without touching the storage logic.
• Multi-position acquisitions use the bioformats2raw layout (one image group per position).

Quick start

// Write
OMEZarrStorageConfig cfg = new OMEZarrStorageConfig()
    .numResolutionLevels(3)               // build a 3-level pyramid automatically
    .compression(Compression.ZSTD)
    .pixelSize(0.1).spatialUnit("micrometer")
    .addAxis(AxisInfo.builder("time").type(DimensionType.TIME).unit("second").build())
    .addAxis(AxisInfo.builder("channel").type(DimensionType.CHANNEL)
        .channels(List.of(Channel.builder("DAPI").color("#0000FF").build())).build());

OMEZarrStorage store = new OMEZarrStorage("/data", "acq", summaryJson, cfg);
Map<String,Object> axes = new HashMap<>();
axes.put("time", 0); axes.put("channel", 0);
store.putImage(pixels /* short[] */, perImageJson, axes,
               false /* rgb */, 16 /* bitDepth */, height, width);
store.finishedWriting();

// Read (works during acquisition too)
OMEZarrImage img = store.getImage(axes);         // full resolution
OMEZarrImage low = store.getImage(axes, 2);      // downsampled x4
store.close();

// Re-open an existing dataset
OMEZarrStorage ds = OMEZarrStorage.load("/data/acq.ome.zarr");

On-disk layout

acq.ome.zarr/
  zarr.json                 # root group: multiscales (single-pos) OR bioformats2raw.layout
  0/ 1/ 2/                  # pyramid level arrays (single-position); each a Zarr v3 array
  ome-metadata.ndjson       # per-image metadata sidecar (append-only)

For multi-position data the root carries bioformats2raw.layout and each position is a numeric sub-group (0/, 1/, …) containing its own 0/ 1/ 2/ level arrays.

Each array is ordered [<non-spatial axes...>, y, x]. Chunks default to one full Y×X plane per non-spatial index, which keeps single-frame streaming writes chunk-aligned (no read-modify-write) and low-latency for concurrent readers.

Concurrency model

A single dedicated writer thread drains a bounded queue; putImage returns a Future<Void> that completes when the image is durably written, and a full queue blocks the producer (back-pressure). Writer-thread failures are surfaced via checkForWritingException(). Reads check the write-pending buffer first (so a just-queued image is immediately visible) and fall back to the atomic on-disk chunks.

Building and testing

Requires JDK 11+ to build (the Maven toolchain) but targets Java 8 bytecode (maven.compiler.release=8), so the jar is usable from Java 8 projects such as Micro-Manager. The source uses no Java 9+ APIs, and all dependencies are Java 8 bytecode.

mvn test

Cross-language conformance

VerificationFixtureTest writes target/verify-fixture.ome.zarr; verify_zarr.py then reads it with the standard zarr-python v3 stack and asserts the OME-NGFF metadata, pyramid, and pixel values (including downsampled levels) are correct:

python verify_zarr.py            # requires: zarr>=3.0, numpy

Limitations (v1)

• Grayscale only (GRAY8/GRAY16/GRAY32). RGB is rejected with a clear error; OME-NGFF has no canonical interleaved-RGB representation.
• Sharding is opt-in. Zarr v3 sharding batches many chunks into one file, which conflicts with frame-by-frame streaming and read-while-write latency, so it is off by default; enable it (via OMEZarrStorageConfig.sharding) for datasets where file count matters more than live-read latency, ideally on lower pyramid levels.
• No within-plane XY tile stitching / mosaic assembly. Multi-field acquisitions are stored as separate positions rather than one stitched array (unlike NDTiff's row/column tiling).