Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OME-NGFF transfer fails #41

Open
pwalczysko opened this issue Jan 27, 2023 · 18 comments
Open

OME-NGFF transfer fails #41

pwalczysko opened this issue Jan 27, 2023 · 18 comments

Comments

@pwalczysko
Copy link
Member

See the uploaded tar.gz

This is from the image https://merge-ci.openmicroscopy.org/web/webclient/?show=image-243767, user-3.

I had another failure on OME-NGFF export as well already.

Happy to investigate deeper, if unclear , cc @will-moore @joshmoore

zarr-small.tar.gz


Using session for user-3@merge-ci-devspace.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: read-only-1
2023-01-27 16:08:39,732 475        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.0
2023-01-27 16:08:39,763 506        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.11.1 revision: 383bac974cd52e83908b54e4769ebb1e0d0673ee date: 1 December 2022
2023-01-27 16:08:39,866 609        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-01-27 16:08:40,403 1146       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2023-01-27 16:08:40,443 1186       [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /Users/pwalczysko/Work/omero-cli-transfer/zarr-small/./user-3_454/Blitz-0-Ice.ThreadPool.Server-22/2022-08/11/10-19-44.733/multifile-Z1.ome.zarr/multifile-Z1.ome.zarr/OME/METADATA.ome.xml
java.io.IOException: '.zgroup' expected but is not readable or missing in store.
	at com.bc.zarr.ZarrGroup.validateGroupToBeOpened(ZarrGroup.java:109)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:102)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:95)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:87)
	at loci.formats.services.JZarrServiceImpl.getGroupAttr(JZarrServiceImpl.java:104)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:164)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.Memoizer.setId(Memoizer.java:662)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
	at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:387)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:355)
	at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:381)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
	at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
	at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
2023-01-27 16:08:40,444 1187       [      main] INFO      ome.formats.importer.ImportCandidates - 6 file(s) parsed into 0 group(s) with 1 call(s) to setId in 33ms. (42ms total) [0 unknowns]
@erickmartins
Copy link
Collaborator

is this on the pack side? can you omero download the same Image?

@will-moore
Copy link
Member

Look like import failed due to missing .zgroup (java.io.IOException: '.zgroup' expected but is not readable or missing in store.)
But having downloaded the sample, I don't see any missing .zgroup files. Not tested functionality

@pwalczysko
Copy link
Member Author

is this on the pack side? can you omero download the same Image?

@erickmartins
The error appears only after the unpack command, the pack goes through without problems and produces the .tar file as attached here.

@erickmartins
Copy link
Collaborator

and if you unzip it, can you run an omero import pointing to the folder where the zarr is?

@pwalczysko
Copy link
Member Author

and if you unzip it, can you run an omero import pointing to the folder where the zarr is?

@erickmartins This fails as below

omero import --depth 10  zarr-small
Using OMERO.java-5.6.6-ice36
Using session for user-3@merge-ci-devspace.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: read-only-1
2023-01-27 16:54:36,002 434        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.0
2023-01-27 16:54:36,029 461        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.11.1 revision: 383bac974cd52e83908b54e4769ebb1e0d0673ee date: 1 December 2022
2023-01-27 16:54:36,117 549        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-01-27 16:54:36,565 997        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2023-01-27 16:54:36,597 1029       [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /Users/pwalczysko/Work/omero-cli-transfer/zarr-small/user-3_454/Blitz-0-Ice.ThreadPool.Server-22/2022-08/11/10-19-44.733/multifile-Z1.ome.zarr/multifile-Z1.ome.zarr/OME/METADATA.ome.xml
java.io.IOException: '.zgroup' expected but is not readable or missing in store.
	at com.bc.zarr.ZarrGroup.validateGroupToBeOpened(ZarrGroup.java:109)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:102)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:95)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:87)
	at loci.formats.services.JZarrServiceImpl.getGroupAttr(JZarrServiceImpl.java:104)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:164)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.Memoizer.setId(Memoizer.java:662)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
	at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:387)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:355)
	at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:381)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
	at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
	at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
2023-01-27 16:54:36,620 1052       [      main] INFO      ome.formats.importer.ImportCandidates - 7 file(s) parsed into 0 group(s) with 2 call(s) to setId in 46ms. (54ms total) [1 unknowns]
2023-01-27 16:54:36,671 1103       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to merge-ci-devspace.openmicroscopy.org:4064
2023-01-27 16:54:37,294 1726       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-01-27 16:54:37,661 2093       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-01-27 16:54:37,689 2121       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2023-01-27 16:54:37,689 2121       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.0
2023-01-27 16:54:37,689 2121       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.9.1
2023-01-27 16:54:37,689 2121       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Mac OS X
2023-01-27 16:54:37,690 2122       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: x86_64
2023-01-27 16:54:37,690 2122       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 10.16
No imports due to errors!

@erickmartins
Copy link
Collaborator

then it's an omero import error, I'd say - I'm just using it :)

@pwalczysko
Copy link
Member Author

pwalczysko commented Jan 30, 2023 •
edited

then it's an omero import error, I'd say - I'm just using it :)

I am afraid I have a counter-example workflow, showing that the problem is either in the pack or in the import command as used by the unpack. I would claim that there are at least 2 problems, as hopefully shown by the workflow below.

  1. Import an ome.zarr file into OMERO -> success, produces a https://merge-ci.openmicroscopy.org/web/webclient/?show=image-256897 (user-3)
  2. Export the freshly imported ome.zarr using OMERO.web -> success, produced a transfer-test.zarr.zip which is attached -> observe that when this is unzipped, it produces a ffolder called multifile-Z1.ome.zarr, i.e. a folder of different name than the downloaded zip
  3. Unzip the transfer-test.zarr.zip and import the resulting folder into OMERO using an import command as per [1] (with the --depth 10 flag)
  4. Observe that Ad 3. above was a success, it produces https://merge-ci.openmicroscopy.org/web/webclient/?show=image-256898 in OMERO
  5. Use the pack command on the image in OMERO Ad 1. -> apparent success
  6. The Ad 5. above produces a test-zarr.tar which is zipped and the test-zarr.tar.gz is attached
  7. Use the unpack command on the test-zarr.tar
  8. Observe that Ad 7 fails with an error similar to OME-NGFF transfer fails #41 (comment) (ie. .zgroup not readable)
  9. Try to unzip the test-zarr.tar locally using the Mac means, which produces a folder test-zarr
  10. Import the folder Ad 9 into OMERO (using the command as per [2], without the --depth 10 flag
  11. Observe that Ad 10 fails, but with a different error [2] than Ad 7, ie. not the .zgroup but the No imports found is the main message, see [2]
  12. Attempt a new import of the test-zarr folder (the result of the pack command from Ad 5), this time adding the --depth 10 flag [3]. Observe that this time, the message is not no imports found anymore, but the .zgroup, ie. similar to Ad 7 (==similar to the unpack error)

[1]
omero import --depth 10 ~/Downloads/multifile-Z1.ome.zarr # - import of the folder downloaded via OMERO.web - note that the name of the unzipped file is different from the downloaded .zip name, which was as in Ad 1 above, transfer-test.zarr.zip

[2] - import of the folder produced by pack command without the depth 10 flag

omero import test-zarr
Using OMERO.java-5.6.6-ice36
Previous session expired for user-3 on merge-ci-devspace.openmicroscopy.org:4064
Server: [merge-ci-devspace.openmicroscopy.org:4064]
Username: [user-3]
Password:
Created session for user-3@merge-ci-devspace.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: read-only-1
2023-01-30 15:51:13,674 452        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.0
2023-01-30 15:51:13,705 483        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.11.1 revision: 383bac974cd52e83908b54e4769ebb1e0d0673ee date: 1 December 2022
2023-01-30 15:51:13,807 585        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-01-30 15:51:14,309 1087       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2023-01-30 15:51:14,338 1116       [      main] INFO      ome.formats.importer.ImportCandidates - 1 file(s) parsed into 0 group(s) with 1 call(s) to setId in 26ms. (30ms total) [1 unknowns]
2023-01-30 15:51:14,400 1178       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to merge-ci-devspace.openmicroscopy.org:4064
2023-01-30 15:51:15,169 1947       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-01-30 15:51:16,722 3500       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-01-30 15:51:16,964 3742       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2023-01-30 15:51:16,964 3742       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.0
2023-01-30 15:51:16,965 3743       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.9.1
2023-01-30 15:51:16,965 3743       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Mac OS X
2023-01-30 15:51:16,965 3743       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: x86_64
2023-01-30 15:51:16,965 3743       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 10.16
No imports found

[3] (import of the folder produced by pack with --depth 10 flag

omero import --depth 10 test-zarr
Using OMERO.java-5.6.6-ice36
Previous session expired for user-3 on merge-ci-devspace.openmicroscopy.org:4064
Server: [merge-ci-devspace.openmicroscopy.org:4064]
Username: [user-3]
Password:
Created session for user-3@merge-ci-devspace.openmicroscopy.org:4064. Idle timeout: 10 min. Current group: read-only-1
2023-01-30 16:02:05,419 445        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.6.0
2023-01-30 16:02:05,449 475        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.11.1 revision: 383bac974cd52e83908b54e4769ebb1e0d0673ee date: 1 December 2022
2023-01-30 16:02:05,560 586        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2023-01-30 16:02:06,062 1088       [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2023-01-30 16:02:06,097 1123       [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /Users/pwalczysko/Work/omero-cli-transfer/test-zarr/user-3_454/Blitz-0-Ice.ThreadPool.Server-22/2023-01/30/15-30-45.337/multifile-Z1.ome.zarr/multifile-Z1.ome.zarr/OME/METADATA.ome.xml
java.io.IOException: '.zgroup' expected but is not readable or missing in store.
	at com.bc.zarr.ZarrGroup.validateGroupToBeOpened(ZarrGroup.java:109)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:102)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:95)
	at com.bc.zarr.ZarrGroup.open(ZarrGroup.java:87)
	at loci.formats.services.JZarrServiceImpl.getGroupAttr(JZarrServiceImpl.java:104)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:164)
	at loci.formats.FormatReader.setId(FormatReader.java:1443)
	at loci.formats.ImageReader.setId(ImageReader.java:849)
	at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at loci.formats.Memoizer.setId(Memoizer.java:662)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
	at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
	at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:387)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:355)
	at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:381)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
	at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
	at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
2023-01-30 16:02:06,115 1141       [      main] INFO      ome.formats.importer.ImportCandidates - 7 file(s) parsed into 0 group(s) with 2 call(s) to setId in 46ms. (54ms total) [1 unknowns]
2023-01-30 16:02:06,176 1202       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to merge-ci-devspace.openmicroscopy.org:4064
2023-01-30 16:02:06,864 1890       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2023-01-30 16:02:07,242 2268       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2023-01-30 16:02:07,269 2295       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2023-01-30 16:02:07,269 2295       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.6.0
2023-01-30 16:02:07,269 2295       [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.9.1
2023-01-30 16:02:07,269 2295       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Mac OS X
2023-01-30 16:02:07,269 2295       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: x86_64
2023-01-30 16:02:07,269 2295       [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 10.16
No imports due to errors!

transfer-test.zarr.zip (file produced by OMERO.web)

test-zarr.tar.gz (file produced by cli-transfer)

Edit: I guess a lot would be revealed by comparing those two zips attached to this comment, which should actually be identical imho. One is produced by ome-cli-transfer, one is produced by OMERO.web export.

@pwalczysko
Copy link
Member Author

Umm, sorry for the naming of the attached zip files, confusingly, the transfer-test.zarr.zip is the one which was NOT probuced by omero-cli-transfer, whereas the test-zarr.tar.gz WAS produced yb omero-cli-transfer.

Thinking about it after re-reading my report, I think we might have problems with naming (which demonstrates in the zgroup not found error) and problems with --depth of scanning, which demonstrates in the no imports found - but I am afraid I do not think that ome-cli-transfer is doing everything correctly here.

@erickmartins
Copy link
Collaborator

erickmartins commented Jan 30, 2023 •
edited

Just double-checking: is the non-pack zip produced by omero export or omero download? If it is omero export, can you try omero download?

@pwalczysko
Copy link
Member Author

Just double-checking: is the non-pack zip produced by omero export or omero download? If it is omero export, can you try omero download?

I think neither, the precise answer is "Done with OMERO.web, the code is:" #41 (comment)

@erickmartins
Copy link
Collaborator

erickmartins commented Jan 30, 2023 •
edited

What does omero export and omero download produce? Can you reimport from those?

What it comes down to is: if omero export and/or omero download produce a structure that can be reimported, I need to see why transfer can't do the same. If either/neither of them can generate a structure that can be reimported, I would rather caveat that in transfer and wait for the fixes there. If the only way right now to get a zarr from OMERO that can be easily reimported is with custom code, I am not going to adapt transfer to that, to be honest. Those fixes should be going to the upstream utilities we're using.

(if you tried to link to code in the previous post, that has not worked)

(well I was typing this while Petr was typing as well. questions answered below)

@pwalczysko
Copy link
Member Author

@erickmartins Further to your question of omero export vs omero download. I am afraid I have tried both commands such as

omero download Image:256897  test-download
... # success
omero import --depth 10 test-download 
...# success

and export

omero export Image:256897 -f test-export
...#success
omero import --depth 10 test-export
...#success

@erickmartins
Copy link
Collaborator

ok, thanks. do they work without the --depth argument?

@pwalczysko
Copy link
Member Author

ok, thanks. do they work without the --depth argument?

Yes, they do work both of them

omero import  test-download
...# success

omero import test-export
...# success

@erickmartins
Copy link
Collaborator

hmmm ok. I'll investigate. omero transfer is just using omero download under the hood, the structure should be the same. thanks for all the testing!

@pwalczysko
Copy link
Member Author

hmmm ok. I'll investigate. omero transfer is just using omero download under the hood, the structure should be the same. thanks for all the testing!

@erickmartins I think I got the cause of the issue. It is a variation of ome/ZarrReader#26 - although the error in ome/ZarrReader#26 and in here is different, the root of the problem is the same - the fact that the "real-to-be-imported" folder xxx.ome.zarr is within another folder, also named xxx.ome.zarr . This enclosing folder naming seems to trip the BF machinery, which looks for the .zgroup in the wrong folder. It is enough to rename the enclosing folder to anything else than .ome.zarr and the import starts to work. See cmds below.

omero transfer pack Image:256897 test-rename-zarr.tar #success
mkdir test-rename-zarr  
tar -xvf test-rename-zarr.tar -C test-rename-zarr
mv test-rename-zarr/user-3_454/Blitz-0-Ice.ThreadPool.Server-22/2023-01/30/15-30-45.337/multifile-Z1.ome.zarr test-rename-zarr/user-3_454/Blitz-0-Ice.ThreadPool.Server-22/2023-01/30/15-30-45.337/multifile-Z1renamed # this is renaming the enclosing folder, note that there is another folder of the same name under it
omero import --depth 10 test-rename-zarr/ #successful import
omero import  test-rename-zarr/ #failure due to depth of scanning, it needs the depth flag, message is No imports found

In summary, the issue could be solved by either

  1. fixing Detection of files not inside the zarr folder if nested into another .zarr folder ZarrReader#26
  2. avoiding the naming of the enclosing folder as .ome.zarr -> note that renaming the image in OMERO prior to running pack will not help, as the name is taken from the original file name
  3. moving the "real" .ome.zarr folder out of the enclosing folder prior to unpack/import

@pwalczysko
Copy link
Member Author

cc @dgault

@pwalczysko pwalczysko mentioned this issue Feb 1, 2023
Open
@erickmartins
Copy link
Collaborator

It is, of course, possible to make an exception for zarrs when it comes to the folder naming conventions, but unless any user has a pressing need for Zarr transfers I think I'd rather wait and see if ZarrReader can deal with that at some point. I'll add a note to README in the future clarifying that Zarr transfers require some moving data around/pointing directly to folders rather than tar/zip files.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@will-moore @erickmartins @pwalczysko