Got only 0s and 1s in segmentation output

[EmiliaCXY]

Hello!

I was running Mesmer to segment whole tumor images (~36k by 42k pixels) on an M60 GPU and I asked for 450G RAM. I have a few questions below and would appreciate any input. (Also I've tried other GPUs and had the same issues.)

So the code I used was segmentation_predictions = app.predict(X_train, image_mpp=0.5).

1. Mesmer was able to finish whole-cell segmentation, however, all predicted cells were labelled with 1s.

Here's a screenshot of how the segmentation mask:

The output of np.unique(segmentation_predictions[0,:,:,0], return_counts = True) was just {0: 1128105591, 1: 433044321}

Also, I got a warning - UserWarning: Only one label was provided to 'remove_small_objects'. Did you mean to use a boolean array?. This is a new warning compared to my other successful runs, so I was wondering if you have any insights about what might have caused all cells to get labelled as 1 and how to address it. I also attached the entire output prints at the end in case any of it is useful.

2 . I got a segmentation fault when I tried to run nuclear segmentation on the same image.

This is the code I used: segmentation_predictions_nuc = app.predict(X_train, image_mpp=0.5, compartment='nuclear') See the end for the entire error message print. Was there a reason why the code was able to finish on whole-cell segmentation but threw errors on nuclear segmentation?

3. I saw the posts #628 and #582 about segmenting large images and agreed that using a series of overlapping patches of 20k by 20k could be a workaround. I was curious if handling large images could be something that Mesmer handles internally?

Thank you so much for your time and efforts!

---

Error log related to Q1 above:

Loading singularity/current
  Loading requirement: squashfs/4.4
2023-02-20 13:20:20.615690: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2023-02-20 13:20:25.815726: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2023-02-20 13:20:25.859800: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:04:00.0 name: Tesla M60 computeCapability: 5.2
coreClock: 1.1775GHz coreCount: 16 deviceMemorySize: 7.94GiB deviceMemoryBandwidth: 149.31GiB/s
2023-02-20 13:20:25.859845: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2023-02-20 13:20:25.935615: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2023-02-20 13:20:25.935680: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2023-02-20 13:20:25.965429: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2023-02-20 13:20:25.988641: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2023-02-20 13:20:26.003061: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2023-02-20 13:20:26.024686: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2023-02-20 13:20:26.026200: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2023-02-20 13:20:26.030492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2023-02-20 13:20:26.030790: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-20 13:20:26.032608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:04:00.0 name: Tesla M60 computeCapability: 5.2
coreClock: 1.1775GHz coreCount: 16 deviceMemorySize: 7.94GiB deviceMemoryBandwidth: 149.31GiB/s
2023-02-20 13:20:26.034793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2023-02-20 13:20:26.036767: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2023-02-20 13:20:29.416226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-02-20 13:20:29.416279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2023-02-20 13:20:29.416288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2023-02-20 13:20:29.419283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7435 MB memory) -> physical GPU (device: 0, name: Tesla M60, pci bus id: 0000:04:00.0, compute capability: 5.2)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:375: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  "The `lr` argument is deprecated, use `learning_rate` instead.")
WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.
2023-02-20 13:28:10.444427: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2023-02-20 13:28:10.464570: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2400045000 Hz
2023-02-20 13:28:11.542577: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2023-02-20 13:28:13.430072: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8100
2023-02-20 13:28:16.164695: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2023-02-20 13:28:18.189487: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
/usr/local/lib/python3.6/dist-packages/deepcell_toolbox/deep_watershed.py:108: UserWarning: h_maxima peak finding algorithm was selected, but the provided image is larger than 5k x 5k pixels.This will lead to slow prediction performance.
  warnings.warn('h_maxima peak finding algorithm was selected, '
/usr/local/lib/python3.6/dist-packages/deepcell_toolbox/deep_watershed.py:194: UserWarning: Only one label was provided to `remove_small_objects`. Did you mean to use a boolean array?
  min_size=small_objects_threshold)

Error log related to Q2 above:

Loading singularity/current
  Loading requirement: squashfs/4.4
2023-02-21 03:57:10.518845: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2023-02-21 03:57:14.819382: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2023-02-21 03:57:14.874436: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:4b:00.0 name: A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.59GiB deviceMemoryBandwidth: 1.41TiB/s
2023-02-21 03:57:14.874471: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2023-02-21 03:57:14.931830: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2023-02-21 03:57:14.931912: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2023-02-21 03:57:14.955492: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2023-02-21 03:57:14.974416: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2023-02-21 03:57:14.985672: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2023-02-21 03:57:15.005573: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2023-02-21 03:57:15.006772: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2023-02-21 03:57:15.010835: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2023-02-21 03:57:15.011103: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-21 03:57:15.012493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:4b:00.0 name: A100-SXM4-40GB computeCapability: 8.0
coreClock: 1.41GHz coreCount: 108 deviceMemorySize: 39.59GiB deviceMemoryBandwidth: 1.41TiB/s
2023-02-21 03:57:15.014643: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2023-02-21 03:57:15.016235: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2023-02-21 03:57:17.108221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-02-21 03:57:17.108251: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2023-02-21 03:57:17.108255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2023-02-21 03:57:17.111858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 38424 MB memory) -> physical GPU (device: 0, name: A100-SXM4-40GB, pci bus id: 0000:4b:00.0, compute capability: 8.0)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:375: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.
  "The `lr` argument is deprecated, use `learning_rate` instead.")
WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.
2023-02-21 04:02:48.376090: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2023-02-21 04:02:48.391906: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3000000000 Hz
2023-02-21 04:02:49.232419: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2023-02-21 04:02:51.351168: I tensorflow/stream_executor/cuda/cuda_dnn.cc:359] Loaded cuDNN version 8100
2023-02-21 04:02:54.135731: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2023-02-21 04:02:56.274770: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
/cm/local/apps/slurm/var/spool/job10514366/slurm_script: line 13: 659449 Segmentation fault      singularity exec --nv --bind /mnt -H /home/chenx5 --cleanenv /mnt/isilon/oldridge_lab/singularity_sif/deepcell-tf_0.10.0-gpu.sif python run_Mesmer_ultivue.py

[msschwartz21]

Hi Emilia!

If I'm reading your logs correctly, it looks like you are running deepcell 0.10.0. Is that right? Could you try running the latest version of deepcell (0.12.4) and let us know if you see the same problems? It would be helpful if you could show us the output of pip list so that we can see what versions of packages you are running.

1. The function to remove_small_objects runs as part of our postprocessing workflow and is reacting to the mask consisting of 0s and 1s. I suspect that something is failing in the process of transforming model outputs into segmentation masks during postprocessing. I'd like to see if the problem is eliminated in the latest release.

2. The final error line /cm/local/apps/slurm/var/spool/job10514366/slurm_script: line 13: 659449 Segmentation fault singularity exec --nv --bind /mnt -H /home/chenx5 --cleanenv /mnt/isilon/oldridge_lab/singularity_sif/deepcell-tf_0.10.0-gpu.sif python run_Mesmer_ultivue.py unfortunately isn't particularly informative. It doesn't look like an error message that we generate in deepcell so I suspect it might be part of the run_mesmer_ultivue.py script. Can you see if you can collect any additional information about what is failing?

[EmiliaCXY]

Hi Morgan! Thank you for the suggestions.

I ran the 0.12.4 version with the same code and same environment but still have the same issues - 1) only 0s and 1s in the whole-cell segmentation and 2) segmentation fault when I tried to do nuclear segmentation. I didn't get the warning about remove_small_objects though. See below for error log.

Regarding the segmentation fault, I unfortunately don't think I could get more information out of it either. I have attached my run_mesmer_ultivue.py below and think the only line that could've caused the error was from the call to Mesmer.

We have also been thinking what could have caused this, just wanted to pass along this observation in case it was of any use. So could it be that certain packages that Mesmer calls use a 32-bit system? Might be a coincidence, but the largest image we were able to run was ~32k by 32k. If each pixel's memory address was stored in 16-bit and we normalize by 2^31, the largest signed number that a 32-bit system could sore, then 2 bytes * 32k * 32k / 2^31 ~= 0.95. In contrast, the current image is 36k by 42k, then 2 * 36k * 42k / 2^31 ~= 1.4

Thanks again! Let us know if there are other tests that we could do to help diagnose the issue.

---

Error log

2023-02-23 10:23:13.051248: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-23 10:23:15.315077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 7434 MB memory:  -> device: 0, name: Tesla M60, pci bus id: 0000:04:00.0, compute capability: 5.2
WARNING:tensorflow:No training configuration found in save file, so the model was *not* compiled. Compile it manually.
2023-02-23 10:30:01.849487: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8100
/cm/local/apps/slurm/var/spool/job10595290/slurm_script: line 14: 2408669 Segmentation fault      singularity exec --nv --bind /mnt -H /home/chenx5 --cleanenv /mnt/isilon/oldridge_lab/singularity_sif/deepcell-tf_0.12.4-gpu.sif python run_Mesmer_ultivue_0.12.4.py

output log

Training Resolution: 0.5 microns per pixel
Starting Cell Segmentation
{0: 1140958872, 1: 420191040}
Starting Nuclear Segmentation

run_Mesmer_ultivue_0.12.4.py

#!/usr/bin/env python
# coding: utf-8

import os
import numpy as np
from skimage import io
from PIL import Image
import pandas as pd
import tifffile
import glob
from deepcell.applications import Mesmer
from deepcell.utils.plot_utils import make_outline_overlay

app = Mesmer()
print('Training Resolution:', app.model_mpp, 'microns per pixel')

inDir = ....
inOmeTiff = ....

im = io.imread(inDir + inOmeTiff) # im.shape is (2, im_height, im_width)
X_train = np.transpose(im, (1,2,0))
X_train = X_train.reshape(1, X_train.shape[0], X_train.shape[1], X_train.shape[2])

### Segment cells by Mesmer and save cellular segmentation mask
print('Starting Cell Segmentation')
segmentation_predictions = app.predict(X_train, image_mpp=0.5)

num, count = np.unique(segmentation_predictions[0,:,:,0], return_counts = True)
num_count = dict(zip(num, count))
print(num_count)

tifffile.imwrite('/mnt/isilon/oldridge_lab/chenx5/ultivue_pilot_run/tiff_process_output/mesmer_output_HGG_Whole_Tumor_Day_1_segmentation_cell_bigtiff_0.12.4.tif',segmentation_predictions[0,:,:,0],bigtiff=True) 

# *** NOTE: We were able to get output from the above line without a segmentation fault ***

### Segment nuclei by Mesmer and save nuclear segmentation mask
print('Starting Nuclear Segmentation')
segmentation_predictions_nuc = app.predict(X_train, image_mpp=0.5, compartment='nuclear')

num, count = np.unique(segmentation_predictions_nuc[0,:,:,0], return_counts = True)
num_count = dict(zip(num, count))
print(num_count)

tifffile.imwrite('/mnt/isilon/oldridge_lab/chenx5/ultivue_pilot_run/tiff_process_output/mesmer_output_HGG_Whole_Tumor_Day_1_segmentation_nuclear_bigtiff_0.12.4.tif',segmentation_predictions_nuc[0,:,:,0],bigtiff=True)

Output of pip list

Package           Version
----------------- ----------
asttokens         2.2.1
attrs             22.2.0
backcall          0.2.0
certifi           2022.12.7
click             8.1.3
click-plugins     1.1.1
cligj             0.7.2
comm              0.1.2
contourpy         1.0.6
cycler            0.11.0
debugpy           1.6.5
decorator         5.1.1
entrypoints       0.4
executing         1.2.0
Fiona             1.9.1
fonttools         4.38.0
geojson           3.0.0
geopandas         0.12.2
imagecodecs       2022.12.24
imageio           2.24.0
ipykernel         6.20.1
ipython           8.8.0
jedi              0.18.2
jupyter_client    7.4.9
jupyter_core      5.1.3
kiwisolver        1.4.4
matplotlib        3.6.3
matplotlib-inline 0.1.6
munch             2.5.0
nest-asyncio      1.5.6
networkx          3.0
numpy             1.24.1
packaging         23.0
pandas            1.5.2
parso             0.8.3
pexpect           4.8.0
pickleshare       0.7.5
Pillow            9.4.0
pip               22.3.1
platformdirs      2.6.2
prompt-toolkit    3.0.36
psutil            5.9.4
ptyprocess        0.7.0
pure-eval         0.2.2
Pygments          2.14.0
pyparsing         3.0.9
pyproj            3.4.1
python-dateutil   2.8.2
pytz              2022.7
PyWavelets        1.4.1
pyzmq             25.0.0
scikit-image      0.19.3
scikit-misc       0.1.4
scipy             1.10.0
seaborn           0.12.2
setuptools        65.6.3
shapely           2.0.1
six               1.16.0
stack-data        0.6.2
tifffile          2022.10.10
tornado           6.2
traitlets         5.8.1
wcwidth           0.2.5
wheel             0.37.1

[rossbar]

Thanks @EmiliaCXY for the updated info:

I ran the 0.12.4 version with the same code and same environment but still have the same issues - 1) only 0s and 1s in the whole-cell segmentation and 2) segmentation fault when I tried to do nuclear segmentation. I didn't get the warning about remove_small_objects though. See below for error log.

Thanks for trying the latest version. There's nothing immediately obvious that points to a specific culprit.

Regarding the segmentation fault, I unfortunately don't think I could get more information out of it either. I have attached my run_mesmer_ultivue.py below and think the only line that could've caused the error was from the call to Mesmer.

Segmentation faults are notoriously tricky to track down "/. In the absence of a self-contained reproducible example, it will be difficult to assist here. If there is e.g. a single image which causes these problems which you are able to share, I may be able to take a closer look. If not, the next thing I would try would be attempting to run the example with a debugger like gdb.

We have also been thinking what could have caused this, just wanted to pass along this observation in case it was of any use.

It's an interesting idea - as long as you're not running on a 32-bit system (which likely isn't even possible, given the dependency tree) this is unlikely to be the issue. Something to keep in mind though!

Output of pip list

Thanks for this - it seems that this is not picking up the correct environment? Note e.g. that deepcell-tf and tensorflow don't appear in the list. What exactly is going on here will depend on how you set up/manage your environment(s), i.e. venv, conda, etc. If you can find & share the info about the environment in which the above script is being run that would be useful.

[ngreenwald]

I would suggest running the model on a very small crop (1024x1024) of your input image to make sure the memory/large image problems can be separated from the 0s and 1s.

I would also run the model in your environment on one image from the example dataset in the Mesmer notebook to further track down where the issue is coming from

[EmiliaCXY]

Thanks @rossbar and @ngreenwald!! We really appreciate it!

Sorry, see below for the pip list output from within the singularity container. We've had other images that we were able to run mesmer successfully, so I think the environment is set up correctly.

We'd be happy to share an image with you! We can share a link with you to download the data from. Are rbarnowski@caltech, nfgreen@stanford, msschwartz@caltech the right emails to reach you?

Thank you for looking into this!

Package                      Version
---------------------------- --------------------
absl-py                      1.0.0
anyio                        3.6.2
argon2-cffi                  21.3.0
argon2-cffi-bindings         21.2.0
arrow                        1.2.3
asttokens                    2.2.1
astunparse                   1.6.3
attrs                        22.1.0
backcall                     0.2.0
beautifulsoup4               4.11.1
bleach                       5.0.1
cachetools                   5.0.0
certifi                      2019.11.28
cffi                         1.15.1
chardet                      3.0.4
click                        8.1.3
click-plugins                1.1.1
cligj                        0.7.2
comm                         0.1.2
contourpy                    1.0.6
cycler                       0.11.0
Cython                       0.29.32
dbus-python                  1.2.16
debugpy                      1.6.5
decorator                    5.1.1
DeepCell                     0.12.4
DeepCell-Toolbox             0.12.0
DeepCell-Tracking            0.6.4
defusedxml                   0.7.1
entrypoints                  0.4
executing                    1.2.0
fastjsonschema               2.16.2
Fiona                        1.9.1
flatbuffers                  2.0
fonttools                    4.38.0
fqdn                         1.5.1
gast                         0.5.3
geojson                      3.0.0
geopandas                    0.12.2
google-auth                  2.6.0
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
grpcio                       1.43.0
h5py                         3.6.0
idna                         2.8
imageio                      2.23.0
importlib-metadata           4.10.1
importlib-resources          5.10.1
ipykernel                    6.20.2
ipython                      8.8.0
ipython-genutils             0.2.0
ipywidgets                   8.0.4
isoduration                  20.11.0
jedi                         0.18.2
Jinja2                       3.1.2
joblib                       1.2.0
jsonpointer                  2.3
jsonschema                   4.17.3
jupyter                      1.0.0
jupyter_client               7.4.9
jupyter-console              6.4.4
jupyter_core                 5.1.3
jupyter-events               0.5.0
jupyter_server               2.0.2
jupyter_server_terminals     0.4.3
jupyterlab-pygments          0.2.2
jupyterlab-widgets           3.0.5
keras                        2.8.0
Keras-Preprocessing          1.1.2
kiwisolver                   1.4.4
libclang                     13.0.0
lxml                         4.9.2
Markdown                     3.3.6
MarkupSafe                   2.1.1
matplotlib                   3.6.2
matplotlib-inline            0.1.6
mistune                      2.0.4
munch                        2.5.0
nbclassic                    0.4.8
nbclient                     0.7.2
nbconvert                    7.2.7
nbformat                     5.7.1
nest-asyncio                 1.5.6
networkx                     2.8.8
notebook                     6.5.2
notebook_shim                0.2.2
numpy                        1.22.1
oauthlib                     3.2.0
opencv-python-headless       4.6.0.66
opt-einsum                   3.3.0
packaging                    22.0
pandas                       1.5.3
pandocfilters                1.5.0
parso                        0.8.3
pexpect                      4.8.0
pickleshare                  0.7.5
Pillow                       9.3.0
pip                          22.3.1
pkgutil_resolve_name         1.3.10
platformdirs                 2.6.2
prometheus-client            0.15.0
prompt-toolkit               3.0.36
protobuf                     3.19.4
psutil                       5.9.4
ptyprocess                   0.7.0
pure-eval                    0.2.2
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycparser                    2.21
pydot                        1.4.2
Pygments                     2.13.0
PyGObject                    3.36.0
pyparsing                    3.0.9
pyproj                       3.4.1
pyrsistent                   0.19.2
python-apt                   2.0.0+ubuntu0.20.4.6
python-dateutil              2.8.2
python-json-logger           2.0.4
pytz                         2022.7.1
PyWavelets                   1.4.1
PyYAML                       6.0
pyzmq                        25.0.0
qtconsole                    5.4.0
QtPy                         2.3.0
requests                     2.22.0
requests-oauthlib            1.3.1
requests-unixsocket          0.2.0
rfc3339-validator            0.1.4
rfc3986-validator            0.1.1
rsa                          4.8
scikit-image                 0.19.3
scikit-learn                 1.2.0
scipy                        1.9.3
Send2Trash                   1.8.0
setuptools                   60.7.0
shapely                      2.0.1
six                          1.14.0
sniffio                      1.3.0
soupsieve                    2.3.2.post1
spektral                     1.0.6
stack-data                   0.6.2
tensorboard                  2.8.0
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow                   2.8.0
tensorflow-addons            0.16.1
tensorflow-io-gcs-filesystem 0.23.1
termcolor                    1.1.0
terminado                    0.17.1
tf-estimator-nightly         2.8.0.dev2021122109
threadpoolctl                3.1.0
tifffile                     2023.2.3
tinycss2                     1.2.1
tornado                      6.2
tqdm                         4.64.1
traitlets                    5.8.1
typeguard                    2.13.3
typing_extensions            4.0.1
uri-template                 1.2.0
urllib3                      1.25.8
wcwidth                      0.2.5
webcolors                    1.12
webencodings                 0.5.1
websocket-client             1.4.2
Werkzeug                     2.0.2
wheel                        0.34.2
widgetsnbextension           4.0.5
wrapt                        1.13.3
zipp                         3.7.0

[EmiliaCXY]

Hi Noah, Ross, and Morgan! Just wanted to update that I ran v0.12.4 on a small patch (2048 * 2048) of the whole slide image and it was able to segment nicely (both whole cell and nuclear), so the environment and the input image should be okay. Below is a screenshot overlaying the nuclear channel input and the whole cell segmentation mask.

[ngreenwald]

And this is a small crop of the same image that was initially giving you the issues? Sounds like it's definitely related to the large image size. Perhaps it's on the verge of a seg fault? The first result is the malformed output, and the second is an actual segfault when you call the model again?

[EmiliaCXY]

Hi @ngreenwald! Yes the segmentation mask above is on a small crop of the same image.

So when I inputted the whole image to mesmer and did whole cell segmentation, the segmentation mask was all 0s and 1s. And then I got a seg fault when I called mesmer again on the same image to do nuclear segmentation.

[ngreenwald]

Yeah, have to admit I'm a bit stumped on this one. We've had people process large images without issues before. Not sure what the root cause is here.

[EmiliaCXY]

Thank you for all the suggestions and thoughts though! We really really appreciate it.

Would you or @rossbar potentially be able to test our image on your server? We can provide the input membrane-nuclear two-channel tiff file that I referenced here and the exact code I used. We have tried processing large images through mesmer on two of our HPC servers and ran into the same issue, so it'd be nice to see if this is reproducible on your side. But no worries if you don't have the bandwidth and we will tile our images as a workaround.

Thanks a lot again!

[ngreenwald]

We're looking into having more seamless support for large images. It'll require some updates, so I would suggest moving forward with tiled processing in the meantime. If you pick large-ish tiles, you shouldn't end up having too many boundaries to worry about. Thanks for bringing this up! If you run into any other issues specific to large images, definitely let us know

[rossbar]

@EmiliaCXY if you have a secure way of sharing the input data that's causing these problems, I'd be happy to take a look.

[EmiliaCXY]

@rossbar Thank you!! We can share the input file via SFTP.

[rossbar]

@EmiliaCXY in the interest of minimizing the number of places my contact info is posted, please refer to my github profile page!