Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Detection of files not inside the zarr folder if nested into another .zarr folder #26

Open
pwalczysko opened this issue Apr 7, 2022 · 7 comments
Open

Detection of files not inside the zarr folder if nested into another .zarr folder #26

pwalczysko opened this issue Apr 7, 2022 · 7 comments

Comments

@pwalczysko
Copy link
Member

pwalczysko commented Apr 7, 2022 •
edited

Note: The problem described below is only occurring if the petr-test folder is nested into another .zarr folder, such as

/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr. If this nesting is removed, the problem does not occurr, and the resulting import does consit of 1 image with 5 z planes as expected.

Have an ome.zarr folder inside a subfolder called petr-test, e.g.

ls -la petr-test/
total 3
dr-xr-xr-x 3 11894 8000 4096 Apr  6 16:59 .
drwxr-xr-x 5 11615 8000 4096 Apr  6 16:56 ..
drwxr-xr-x 4 11894 8000 4096 Apr  6 16:56 multifile-Z1.ome.zarr

In this case, the importer detects the zarr files to import as follows:

omero import -f multifile-Z1.ome.zarr/
2022-04-07 13:43:35,972 233        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.9-SNAPSHOT
2022-04-07 13:43:35,990 251        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.9.1-SNAPSHOT revision: 17f2529242af94162bfaff168c25746f39ca04ad date: 7 April 2022
2022-04-07 13:43:36,055 316        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-04-07 13:43:36,479 740        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2022-04-07 13:43:37,078 1339       [      main] INFO      ome.formats.importer.ImportCandidates - 6 file(s) parsed into 1 group(s) with 1 call(s) to setId in 588ms. (599ms total) [0 unknowns]
#======================================
# Group: /uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/OME/METADATA.ome.xml SPW: false Reader: loci.formats.in.ZarrReader
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/OME/METADATA.ome.xml
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/.zattrs
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/.zattrs
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/.zarray
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/3/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/2/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/4/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/1/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/0/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/.zgroup
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/.zgroup
/uod/idr-scratch/test.zarr/.zattrs
/uod/idr-scratch/test.zarr/OME/METADATA.ome.xml
/uod/idr-scratch/test.zarr/0/.zattrs
/uod/idr-scratch/test.zarr/0/1/.zarray
/uod/idr-scratch/test.zarr/0/1/0/0/0/0/0
/uod/idr-scratch/test.zarr/0/0/.zarray
/uod/idr-scratch/test.zarr/0/0/0/0/0/0/0
/uod/idr-scratch/test.zarr/0/.zgroup
/uod/idr-scratch/test.zarr/.zgroup

Now, have following 5 files inside the petr-test folder as well next to the ome.zarr folder (not inside the ome.zarr) such as

ls -la petr-test/
total 6
dr-xr-xr-x 3 11894 8000 4096 Apr  7 14:49 .
drwxr-xr-x 5 11615 8000 4096 Apr  6 16:56 ..
-rwxr-xr-x 1 11894 8000 2773 Jun 23  2016 multifile-Z1.ome.tiff
drwxr-xr-x 4 11894 8000 4096 Apr  6 16:56 multifile-Z1.ome.zarr
-rwxr-xr-x 1 11894 8000 1753 Jun 22  2016 multifile-Z2.ome.tiff
-rwxr-xr-x 1 11894 8000 1753 Jun 22  2016 multifile-Z3.ome.tiff
-rwxr-xr-x 1 11894 8000 1753 Jun 22  2016 multifile-Z4.ome.tiff
-rwxr-xr-x 1 11894 8000 1753 Jun 22  2016 multifile-Z5.ome.tiff

The importer is detecting the ome.tiff files as well, although pointed only and directly at the ome.zarr

omero import -f multifile-Z1.ome.zarr/
2022-04-07 14:52:20,506 241        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.9-SNAPSHOT
2022-04-07 14:52:20,525 260        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.9.1-SNAPSHOT revision: 17f2529242af94162bfaff168c25746f39ca04ad date: 7 April 2022
2022-04-07 14:52:20,569 304        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-04-07 14:52:20,951 686        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2022-04-07 14:52:21,558 1293       [      main] INFO      ome.formats.importer.ImportCandidates - 6 file(s) parsed into 1 group(s) with 1 call(s) to setId in 598ms. (607ms total) [0 unknowns]
#======================================
# Group: /uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/OME/METADATA.ome.xml SPW: false Reader: loci.formats.in.ZarrReader
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/OME/METADATA.ome.xml
/uod/idr-scratch/test.zarr/petr-test/multifile-Z3.ome.tiff
/uod/idr-scratch/test.zarr/petr-test/multifile-Z5.ome.tiff
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/.zattrs
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/.zattrs
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/.zarray
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/3/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/2/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/4/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/1/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/0/0/0/0/0/0
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/0/.zgroup
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.zarr/.zgroup
/uod/idr-scratch/test.zarr/petr-test/multifile-Z1.ome.tiff
/uod/idr-scratch/test.zarr/petr-test/multifile-Z2.ome.tiff
/uod/idr-scratch/test.zarr/petr-test/multifile-Z4.ome.tiff
/uod/idr-scratch/test.zarr/.zattrs
/uod/idr-scratch/test.zarr/OME/METADATA.ome.xml
/uod/idr-scratch/test.zarr/0/.zattrs
/uod/idr-scratch/test.zarr/0/1/.zarray
/uod/idr-scratch/test.zarr/0/1/0/0/0/0/0
/uod/idr-scratch/test.zarr/0/0/.zarray
/uod/idr-scratch/test.zarr/0/0/0/0/0/0/0
/uod/idr-scratch/test.zarr/0/.zgroup
/uod/idr-scratch/test.zarr/.zgroup
@pwalczysko pwalczysko changed the title Wrong detection of files belonging to the zarr Detection of files not inside the zarr folder Apr 7, 2022
@pwalczysko
Copy link
Member Author

cc @sbesson @joshmoore cf. discussion this morning

@joshmoore
Copy link
Member

@pwalczysko: can you try this in a subdirectory that has no parents ending in .zarr? I wonder if logic is going up the stack to find a .z file.

@sbesson
Copy link
Member

sbesson commented Apr 7, 2022

I was about to say part of the concern is redundant with #25 around defining precisely how much should be detected/discovered by the reader.
In this case, things are further complicated by the fact the output of a bioformats2raw execution is nested within the output of a format bioformats2raw execution. Sounds like the kind of edge case situations that we don't want to support (and the reader should probably fail accordingly)

@pwalczysko
Copy link
Member Author

pwalczysko commented Apr 7, 2022 •
edited 

[pwalczysko@idr2-slot3 idr-scratch]$ pwd
/uod/idr-scratch
[pwalczysko@idr2-slot3 idr-scratch]$ sudo mv test.zarr/petr-test/ .
[sudo] password for pwalczysko: 
mv: cannot move ‘test.zarr/petr-test/’ to ‘./petr-test’: Permission denied

I think I am starting to understand - but see above, the options are limited to have a better situation, not sure why admin cannot move the folder.

Edit:
As indicated, managed to do the above move whilst being on idr0-slot3. Not sure why it works there, but not on idr2-slot3``

@pwalczysko
Copy link
Member Author

Indeed, the move of the petr-test out of the test.zarr directory solved the issue

ls
multifile-Z1.ome.tiff  multifile-Z2.ome.tiff  multifile-Z4.ome.tiff
multifile-Z1.ome.zarr  multifile-Z3.ome.tiff  multifile-Z5.ome.tiff
(.venv3) bash-4.2$ omero import -f multifile-Z1.ome.zarr/ 
2022-04-07 15:24:49,350 230        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.9-SNAPSHOT
2022-04-07 15:24:49,369 249        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.9.1-SNAPSHOT revision: 17f2529242af94162bfaff168c25746f39ca04ad date: 7 April 2022
2022-04-07 15:24:49,414 294        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-04-07 15:24:49,787 667        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 4 Metadata Level: MINIMUM
2022-04-07 15:24:50,298 1178       [      main] INFO      ome.formats.importer.ImportCandidates - 6 file(s) parsed into 1 group(s) with 1 call(s) to setId in 502ms. (511ms total) [0 unknowns]
#======================================
# Group: /uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/OME/METADATA.ome.xml SPW: false Reader: loci.formats.in.ZarrReader
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/OME/METADATA.ome.xml
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/.zattrs
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/.zattrs
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/.zarray
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/3/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/2/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/4/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/1/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/0/0/0/0/0/0
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/0/.zgroup
/uod/idr-scratch/petr-test/multifile-Z1.ome.zarr/.zgroup

@pwalczysko
Copy link
Member Author

The removal of the petr-test out of the test.zarr also magically solved #25

@pwalczysko pwalczysko mentioned this issue Apr 7, 2022
Open
@pwalczysko pwalczysko changed the title Detection of files not inside the zarr folder Detection of files not inside the zarr folder if nested into another .zarr folder Apr 7, 2022
@pwalczysko pwalczysko mentioned this issue Feb 1, 2023
Open
@pwalczysko
Copy link
Member Author

This problem is resurfacing in ome/omero-cli-transfer#41 (comment) - I would claim that it shows that the issue here is actually caused by the naming of the enclosing folder, and it is less marginal (as shown by the fact that the omero-cli-transfer is hitting it) than previously judged

cc @sbesson @dgault

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@joshmoore @sbesson @pwalczysko