Support export of Fileset:ID #123

will-moore · 2022-07-07T13:43:41Z

Fixes #122

To test: e.g. on merge-ci (user-3):

$ omero zarr export Fileset:115357 --output="test_fileset"

Will create a test_fileset directory with images /0, /1 etc in it.
Directory will contain .zattrs and /OME/METADATA.ome.xml as defined in ome/ngff#112

If directory exists, then test_fileset(1) directory will be created etc.

Support for other multi-image export to the same format NOT included yet. Could come in a follow-up PR e.g:

$ omero zarr export Image:10,11,12 --output="my_images"
$ omero zarr export Dataset:123 --output="my_images"

(would need different object parameter type since current ProxyStringType doesn't support multiple IDs).

NB: to get the Fileset ID from an Image in webclient, see ome/omero-web#385

joshmoore

Looking at the code & the description:

(would need different object parameter type since current ProxyStringType doesn't support multiple IDs).

Could try to add support, though that might be ProxiesStringType :)

I guess we might want to expose this in the UI a bit better?

👍 Considering there are CLI operations on it, that'd be my vote.

joshmoore · 2022-07-08T10:21:31Z

src/omero_zarr/util.py

@@ -112,3 +122,55 @@ def marshal_transformations(
        zooms["y"] = zooms["y"] * multiscales_zoom

    return transformations
+
+
+def get_minimum_image_ome_xml(images: List[ImageWrapper]) -> str:


Might be worth considering ome_model or ome_types for this? cc: @seb

Using ome-types in 553b35f

will-moore · 2022-07-22T11:06:54Z

Webclient PR to show Fileset ID: ome/omero-web#385

pwalczysko

Please address ome/omero-web#385 (comment)

pwalczysko

Sorry, igoner my review, it was not on the correct PR.#123 (review)

joshmoore · 2022-08-02T14:43:23Z

@pwalczysko, but do you also want to give this PR a try? 😉

pwalczysko · 2022-08-03T09:37:44Z

(cli) pwalczysko@ls31618~/Work/omero-cli-zarr$ pip install .
Processing /Users/pwalczysko/Work/omero-cli-zarr
Requirement already satisfied: omero-py>=5.6.0 in /Users/pwalczysko/opt/anaconda3/envs/cli/lib/python3.6/site-packages (from omero-cli-zarr==0.5.1.dev4+g553b35f) (5.9.1)
Collecting ome-zarr>=0.5.0
  Using cached ome_zarr-0.5.2-py3-none-any.whl (32 kB)
ERROR: Could not find a version that satisfies the requirement ome-types (from omero-cli-zarr)
ERROR: No matching distribution found for ome-types

will-moore · 2022-08-03T10:07:30Z

I couldn't reproduce that error.
If I created a clean env using the omero-py docs conda create -n myenv -c conda-forge python=3.8 omero-py then installed omero-cli-zarr with pip install . this worked fine.
If I removed the omero-py from the conda create command then installing omero-cli-zarr fails with ERROR: Failed building wheel for zeroc-ice, which seems to be unrelated to this PR (I'll create another issue)...

But in either case I didn't see the error above. @pwalczysko can you reproduce on a clean conda env?

pwalczysko · 2022-08-03T10:13:20Z

I couldn't reproduce that error. If I created a clean env using the omero-py docs conda create -n myenv -c conda-forge python=3.8 omero-py then installed omero-cli-zarr with pip install . this worked fine. If I removed the omero-py from the conda create command then installing omero-cli-zarr fails with ERROR: Failed building wheel for zeroc-ice, which seems to be unrelated to this PR (I'll create another issue)...

But in either case I didn't see the error above. @pwalczysko can you reproduce on a clean conda env?

I think that the discrepancy is in the python version, my envs have python 3.6. Will try

pwalczysko · 2022-08-03T10:28:20Z

Running a
conda create -n myenv -c conda-forge python=3.8 #leaving out the omeropy bit
conda activate myenv
pip install .

gives me a following error @will-moore

...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/pwalczysko/opt/anaconda3/envs/testzarr/include -fPIC -O2 -isystem /Users/pwalczysko/opt/anaconda3/envs/testzarr/include -arch x86_64 -DICE_STATIC_LIBS -Isrc -Isrc/ice/cpp/include -Isrc/ice/cpp/src -I/Users/pwalczysko/opt/anaconda3/envs/testzarr/include/python3.8 -c src/ice/cpp/src/IceSSL/AcceptorI.cpp -o build/temp.macosx-10.9-x86_64-cpython-38/src/ice/cpp/src/IceSSL/AcceptorI.o -w
      In file included from src/ice/cpp/src/IceSSL/AcceptorI.cpp:14:
      In file included from src/ice/cpp/src/IceSSL/OpenSSLTransceiverI.h:14:
      In file included from src/ice/cpp/src/IceSSL/Util.h:23:
      In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:27:
      /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecBase.h:88:33: error: typedef redefinition with different types ('struct __SecCertificate' vs 'OpaqueSecCertificateRef')
      typedef struct __SecCertificate OpaqueSecCertificateRef;
                                      ^
      src/ice/cpp/include/IceSSL/Plugin.h:67:8: note: previous definition is here
      struct OpaqueSecCertificateRef;
             ^
      In file included from src/ice/cpp/src/IceSSL/AcceptorI.cpp:14:
      In file included from src/ice/cpp/src/IceSSL/OpenSSLTransceiverI.h:14:
      In file included from src/ice/cpp/src/IceSSL/Util.h:23:
      In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:27:
      /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks/Security.framework/Headers/SecBase.h:111:25: error: typedef redefinition with different types ('struct __SecKey' vs 'OpaqueSecKeyRef')
      typedef struct __SecKey OpaqueSecKeyRef;
                              ^
      src/ice/cpp/include/IceSSL/Plugin.h:73:8: note: previous definition is here
      struct OpaqueSecKeyRef;
             ^
      2 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> zeroc-ice

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

will-moore · 2022-08-03T10:29:03Z

Issue created at #127

will-moore · 2022-08-03T10:31:08Z

@pwalczysko I don't see ERROR: No matching distribution found for ome-types there? Do you know how you got that (since that seems more relevant to this PR)?

pwalczysko · 2022-08-03T10:36:42Z

@pwalczysko I don't see ERROR: No matching distribution found for ome-types there? Do you know how you got that (since that seems more relevant to this PR)?

conda create -n test2 python=3.6
conda activate test2
pip install .

elicits the error below @will-moore (note python version in the above command)

Processing /Users/pwalczysko/Work/omero-cli-zarr
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Collecting omero-py>=5.6.0
  Using cached omero_py-5.11.2-py3-none-any.whl (2.8 MB)
Collecting ome-zarr>=0.5.0
  Using cached ome_zarr-0.5.2-py3-none-any.whl (32 kB)
ERROR: Could not find a version that satisfies the requirement ome-types (from omero-cli-zarr) (from versions: none)
ERROR: No matching distribution found for ome-types

will-moore · 2022-08-03T11:38:04Z

Ah - OK, thanks. So the use of python-3.6 should also be handled by better install instructions: #127

If you install omero-py first (as describe on omero-py README, which includes python 3.8) are you able to install omero-cli-zarr OK and test?

pwalczysko · 2022-08-03T11:43:06Z

If you install omero-py first (as describe on omero-py README, which includes python 3.8) are you able to install omero-cli-zarr OK and test?

Yes, this is what I did and it works.

pwalczysko · 2022-08-03T11:49:39Z

Is this functionality supposed to produce real ome.zarr images with pixeldata ?

See https://merge-ci.openmicroscopy.org/web/webclient/?show=image-243571 - user-3, after reimport into OMERO, all values reported as 0, the original image is the https://merge-ci.openmicroscopy.org/web/webclient/?show=image-200515

The same is repeating with any image I tried up till now.

will-moore · 2022-08-03T12:34:54Z

@pwalczysko yes, it is supposed to!

omero zarr export Fileset:115357 --output="test_reimport"

// reimport using latest OMERO release
echo $OMERODIR 
/Users/wmoore/Desktop/OMERO.server-5.6.5-ice36-b233/

omero import --depth=10 test_reimport/
...
Image:243573,243574,243575
Other imported objects:
Fileset:128463

==> Summary
1 file uploaded, 1 fileset created, 3 images imported, 0 errors in 0:00:03.459

It looks like the data chunks haven't been uploaded.
And indeed, these images appear to have all pixel values of 0.

Although they appear OK in napari:

napari --plugin napari-ome-zarr test_reimport/0/

Trying to export and re-import a single image:

$ omero zarr export Image:206965

$ omero import --depth=10 206965.zarr
...
// I see upload of all chunks etc...

2022-08-03 13:22:11,446 16144      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1488889
2022-08-03 13:22:11,641 16339      [l.Client-0] ERROR     ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /Users/wmoore/Desktop/ZARR/data/206965.zarr/0/0/0
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonKey
	at com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector.hasAsKey(JacksonAnnotationIntrospector.java:1080)
	at com.fasterxml.jackson.databind.introspect.POJOPropertiesCollector._addFields(POJOPropertiesCollector.java:495)

==> Summary
10 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:09.060

Trying to import a single image from the Fileset exported above...

omero import --depth=10 test_reimport/0
...
No imports found

Seems that Bio-Formats doesn't recognise this as an OME-NGFF image without .zarr...
Try to rename... get same error as above...

mv test_reimport/0 test_reimport/0.zarr
omero import --depth=10 test_reimport/0.zarr/

2022-08-03 13:32:18,262 16364      [2-thread-1] INFO   ormats.importer.cli.LoggingImportMonitor - IMPORT_STARTED Logfile: 1488933
2022-08-03 13:32:18,459 16561      [  Thread-7] ERROR     ome.formats.importer.cli.ErrorHandler - INTERNAL_EXCEPTION: /Users/wmoore/Desktop/ZARR/data/test_reimport/0.zarr/0/0/0
java.lang.RuntimeException: Failure response on import!
Category: ::omero::grid::ImportRequest
Name: error-on-init
Parameters: {stacktrace=java.lang.NoClassDefFoundError: com/fasterxml/jackson/annotation/JsonKey

==> Summary
10 files uploaded, 0 filesets created, 0 images imported, 1 error in 0:00:09.335

To summarise, import either fails to import the chunks (and you end up with all pixel values of 0) or if chunks are uploaded then the import fails for another reason.

cc @dgault any ideas what's going on here?

pwalczysko · 2022-08-03T12:46:36Z

Thank you @will-moore for the explanation and follow-up testing. From my side, the functionality in this PR works as described, but the reimport of the ome.zarr into OMERO fails as detailed in #123 (comment) - I think this PR is okay, but maybe we should wait with the merging to find out what must be done to fix the complete export -> reimport workflow ?

will-moore · 2022-08-03T13:20:27Z

Trying to import the .zattrs file directly (as suggested by @pwalczysko ):

$ omero import --depth=10 test_reimport/0.zarr/.zattrs
No imports found

pwalczysko · 2022-08-03T13:34:45Z

(.venv3) bash-4.2$ omero import -f  --depth 10 test_real/
2022-08-03 13:32:12,471 201        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.5.13-SNAPSHOT
2022-08-03 13:32:12,488 218        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 6.10.1-SNAPSHOT revision: f45c900610fd6a1c463ec4e21f45d809a38981b6 date: 3 August 2022
2022-08-03 13:32:12,528 258        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2022-08-03 13:32:12,863 593        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2022-08-03 13:32:28,537 16267      [      main] INFO      ome.formats.importer.ImportCandidates - 1028 file(s) parsed into 1 group(s) with 1021 call(s) to setId in 14209ms. (15674ms total) [1020 unknowns]
#======================================
# Group: /uod/idr-scratch/petr-test/test_real/OME/METADATA.ome.xml SPW: false Reader: loci.formats.in.OMEXMLReader
/uod/idr-scratch/petr-test/test_real/OME/METADATA.ome.xml

@will-moore @dgault further to #123 (comment) - I do not think that the correct reader is used, see above.

joshmoore · 2022-08-03T14:34:09Z

@pwalczysko #123 (comment) Yes, this is what I did and it works.

(Side note: internally and externally we should always start from conda create -n test2 -c conda-forge omero-py in order to correctly install Ice)

imagesc-bot · 2023-08-07T08:59:28Z

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/export-large-images-from-omero-via-web/84652/3

snoopycrimecop · 2023-12-16T00:51:44Z

Conflicting PR. Removed from build OMERO-plugins-push#343. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- src/omero_zarr/raw_pixels.py
- .isort.cfg

--conflicts

snoopycrimecop · 2023-12-17T00:54:05Z

Conflicting PR. Removed from build OMERO-plugins-push#344. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2023-12-18T00:52:58Z

Conflicting PR. Removed from build OMERO-plugins-push#345. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- src/omero_zarr/raw_pixels.py
- .isort.cfg

--conflicts

snoopycrimecop · 2023-12-20T00:50:23Z

Conflicting PR. Removed from build OMERO-plugins-push#346. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2023-12-21T00:51:32Z

Conflicting PR. Removed from build OMERO-plugins-push#347. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- src/omero_zarr/raw_pixels.py
- .isort.cfg

--conflicts

snoopycrimecop · 2023-12-22T00:51:54Z

Conflicting PR. Removed from build OMERO-plugins-push#348. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- src/omero_zarr/raw_pixels.py
- .isort.cfg

--conflicts

snoopycrimecop · 2023-12-23T00:52:23Z

Conflicting PR. Removed from build OMERO-plugins-push#349. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2023-12-24T00:52:04Z

Conflicting PR. Removed from build OMERO-plugins-push#350. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2023-12-25T00:54:34Z

Conflicting PR. Removed from build OMERO-plugins-push#351. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2023-12-27T00:52:47Z

Conflicting PR. Removed from build OMERO-plugins-push#352. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2023-12-28T00:52:15Z

Conflicting PR. Removed from build OMERO-plugins-push#353. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- src/omero_zarr/raw_pixels.py
- .isort.cfg

--conflicts

snoopycrimecop · 2023-12-29T00:51:31Z

Conflicting PR. Removed from build OMERO-plugins-push#354. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2023-12-30T00:51:04Z

Conflicting PR. Removed from build OMERO-plugins-push#355. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2023-12-31T00:53:11Z

Conflicting PR. Removed from build OMERO-plugins-push#356. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- src/omero_zarr/raw_pixels.py
- .isort.cfg

--conflicts

snoopycrimecop · 2024-01-01T00:54:33Z

Conflicting PR. Removed from build OMERO-plugins-push#357. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- src/omero_zarr/raw_pixels.py
- .isort.cfg

--conflicts

snoopycrimecop · 2024-01-02T00:51:05Z

Conflicting PR. Removed from build OMERO-plugins-push#358. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

snoopycrimecop · 2024-01-03T00:52:03Z

Conflicting PR. Removed from build OMERO-plugins-push#359. See the console output for more details.
Possible conflicts:

PR Zarr export tests #102 will-moore 'Zarr export tests'
- src/omero_zarr/raw_pixels.py
Upstream changes
- .isort.cfg
- src/omero_zarr/raw_pixels.py

--conflicts

will-moore added 2 commits July 7, 2022 14:36

Support export of Fileset:ID

5966edd

Export .zattrs and OME/METADATA.ome.xml

9492218

will-moore marked this pull request as ready for review July 7, 2022 14:34

joshmoore reviewed Jul 8, 2022

View reviewed changes

will-moore mentioned this pull request Jul 22, 2022

Show fileset ID ome/omero-web#385

Merged

Use ome-types for OME.xml generation

553b35f

pwalczysko suggested changes Aug 2, 2022

View reviewed changes

pwalczysko reviewed Aug 2, 2022

View reviewed changes

will-moore mentioned this pull request Aug 3, 2022

Installation instructions #127

Open

snoopycrimecop mentioned this pull request Jan 10, 2024

scm version fix #159

Merged

Support export of Fileset:ID #123

Are you sure you want to change the base?

Support export of Fileset:ID #123

Conversation

will-moore commented Jul 7, 2022 • edited

joshmoore left a comment

Choose a reason for hiding this comment

joshmoore Jul 8, 2022

Choose a reason for hiding this comment

will-moore Aug 2, 2022

Choose a reason for hiding this comment

will-moore commented Jul 22, 2022

pwalczysko left a comment

Choose a reason for hiding this comment

pwalczysko left a comment

Choose a reason for hiding this comment

joshmoore commented Aug 2, 2022

pwalczysko commented Aug 3, 2022

will-moore commented Aug 3, 2022

pwalczysko commented Aug 3, 2022

pwalczysko commented Aug 3, 2022 • edited

will-moore commented Aug 3, 2022

will-moore commented Aug 3, 2022

pwalczysko commented Aug 3, 2022

will-moore commented Aug 3, 2022

pwalczysko commented Aug 3, 2022

pwalczysko commented Aug 3, 2022

will-moore commented Aug 3, 2022

pwalczysko commented Aug 3, 2022

will-moore commented Aug 3, 2022

pwalczysko commented Aug 3, 2022

joshmoore commented Aug 3, 2022

imagesc-bot commented Aug 7, 2023

snoopycrimecop commented Dec 16, 2023

snoopycrimecop commented Dec 17, 2023

snoopycrimecop commented Dec 18, 2023

snoopycrimecop commented Dec 20, 2023

snoopycrimecop commented Dec 21, 2023

snoopycrimecop commented Dec 22, 2023

snoopycrimecop commented Dec 23, 2023

snoopycrimecop commented Dec 24, 2023

snoopycrimecop commented Dec 25, 2023

snoopycrimecop commented Dec 27, 2023

snoopycrimecop commented Dec 28, 2023

snoopycrimecop commented Dec 29, 2023

snoopycrimecop commented Dec 30, 2023

snoopycrimecop commented Dec 31, 2023

snoopycrimecop commented Jan 1, 2024

snoopycrimecop commented Jan 2, 2024

snoopycrimecop commented Jan 3, 2024

will-moore commented Jul 7, 2022 •

edited

pwalczysko commented Aug 3, 2022 •

edited