Concrete example accessing Zarr store on S3?

[rschmunk]

The documentation gives a fair amount of info how to set up an address for accessing a Zarr store on, e.g., AWS S3. However, there is no example given there of doing so for an existing such store.

I have been trying to load Zarr stores from a couple buckets on AWS S3, and absolutely cannot figure out what use as the "super/long/key" in the constructed address.

Examples: one (see top right bucket link) and two (bottom right bucket link).

Someone I've been working indicate she tried accessing the second example using Python code, and it was a nightmare. But until I can NJ's (CDM)S3 code to stop throwing exceptions as I guess at what key to use, I can't respond to that.

[haileyajohnson]

Since Zarr support is so new and relatively experimental, we have not put together any tutorials as of yet. You can look through our tests for some trivial examples, e.g. here

For the first example you provide, the access URI would be: cdms3:power-analysis-ready-datastore?{top_folder_name}#delimiter=/
Where {top_folder_name} would be something like "power_901_constants.zarr/"

For the second example...I'm not sure our implementation can currently handle an s3 zarr dataset structured like that, since the "top level" in this case is the bucket itself. It should be able to read subgroups, such as cdms3:hrrrzarr?grid/#delimiter=/, but to read the whole dataset with the current implementation you would need to restructure the bucket so that it contains a top level object (e.g. hrrrzarr.zarr/) rather than using the the whole bucket as the dataset.

[haileyajohnson]

Are you including the Zarr module in your build? cdm-zarr

[rschmunk]

@haileyajohnson, Ah, good catch. I was using a different build of the netcdfAll jar than I thought I was. That said, some progress but still no joy.

Trying to open cdms3:unidata-zarr-test-data?zarr_test_data.zarr/#delimiter=/ fails with an NPE in ZarrHeader.makeVariable.

I have been able to open cdms3://aws/power-analysis-ready-datastore?power_901_constants.zarr/#delimiter=/, but it's painfully slow. After about 4 minutes 30 second of waiting, I get a string of 18-20 severe error msgs from ZarrHeader.DelayedVarMaker.processItem, after 5-20 seconds the Zarr store has been acquired and I can get a list of the contents. Trying to extract data does not work, although the reasons differ whether I am trying to create a 1D or 2D plot.

The Zarr stores in the power analysis bucket look much, much too big to test with, so I'll have to cast about and see if I can find something small on AWS.

[rschmunk]

@haileyajohnson, "Are you including the Zarr module in your build?" Which reminds me...

If I want to build a netcdfAll with the optional Zarr code included, I have to start from a SNAPSHOT from 8 months ago, because anything more recent... well, #1045, alas.

[haileyajohnson]

@rschmunk thanks for the reminder, I was working on that and got sidetracked. Unfortunately this week is really busy and I'm not going to be able to work on anything Zarr related until at least next week :/

We definitely still need to do some performance testing and improvement on S3 access as well, but again, we don't have the bandwidth right now.

[rschmunk]

@haileyajohnson, No rush. But I just wanted to make sure it was still on your radar. My main concern is being able to use a current copy of the NJ 5.x code when working with Zarr rather than keep going back to an olde version. And as the Zarr code still seems a bit flaky, it's not like I'm in a hurry to push out a copy of my app that uses it.

Related to that flakiness, I added some timing diagnostic to my code when acquiring a Zarr store on S3. I found that calling NetcdfFiles.open("cdms3://aws/power-analysis-ready-datastore?power_901_constants.zarr/#delimiter=/");
it took 296 seconds from call to return.

The remaining 19-20 seconds btw that finishing and when the dataset info appears in the Panoply display is taken up by finding the store's child variables and their metadata.