double free or corruption (fasttop) with tomopy.find_center

[jcesardasilva]

I am trying to follow the examples described in the Tomopy doc, but when I try to run:
rot_center = tomopy.find_center(proj, theta, init=290, ind=0, tol=0.5)
I get:

*** Error in `/pyvenv_local/bin/python2': double free or corruption (fasttop): 0x00007fd10c014300 ***
Aborted

What could be the reason?

[dgursoy]

You use ‘ind=0’. Have you checked if this slice has ‘nan’s? Usually a slice at the middle, where the sample is expected to be is preferred.

[mkutsal]

Hi, I have encountered the same issue while trying to use tomopy on examples from the tomobank. I have tried both command line and in-python options for any reconstruction, finding center functionalities. The scripts initialize normally, but fails with a similar error message as above. I'm copying an example of the terminal output for center finding:

(tomopy) gpu2-1404:tmp_14_days/Kutsal/tomobank_tomo_00056 % tomopy find_center --file_name=tomo_00056.h5
astropy module not found
2021-11-04 13:54:31,531 - Saving log at /home/esrf/kutsal/logs/tomopy_2021-11-04_13_54_31.log
2021-11-04 13:54:31,531 - find center start
2021-11-04 13:54:31,532 - Found: ['tomo_00056.h5']
2021-11-04 13:54:31,532 - Determining the rotation axis location
2021-11-04 13:54:31,532 - *** calculating automatic center
2021-11-04 13:54:31,533 - *** auto pixel size reading
2021-11-04 13:54:31,533 - *** *** OFF
2021-11-04 13:54:31,534 - *** auto reading filter configuration
2021-11-04 13:54:31,534 - *** *** standard
2021-11-04 13:54:31,534 - *** *** skip finding exposure ratio
2021-11-04 13:54:31,534 - *** loading a stardard data set: tomo_00056.h5
2021-11-04 13:54:31,557 - *** tomo_00056.h5 is a valid dx file format
2021-11-04 13:54:31,557 - *** median filter flat images
2021-11-04 13:54:31,558 - *** median filter dark images
2021-11-04 13:54:31,558 - *** correcting for blocked view data collection
2021-11-04 13:54:31,558 - *** *** OFF
2021-11-04 13:54:31,558 - *** binning
2021-11-04 13:54:31,559 - *** *** OFF
2021-11-04 13:54:31,559 - *** rotation center: -1.000000
2021-11-04 13:54:31,559 - *** zinger removal
2021-11-04 13:54:31,559 - *** *** OFF
2021-11-04 13:54:31,559 - *** normalization
*** Error in `/home/esrf/kutsal/anaconda3/envs/tomopy/bin/python': double free or corruption (!prev): 0x000055e341df19e0 ***
Aborted

I am using tomopy version 2.0 on a conda environment (python 3.9) in ESRF's rnice cluster (should be Debian 8.11 if I'm not mistaken). Thanks in advance!

[carterbox]

@mkutsal, there is no tomopy version 2. The latest is 1.10.4. Could you provide a minimal example that causes the error?

[mkutsal]

@carterbox Hi, thanks for your reply. I have installed tomopy (and tomopy-cli) through conda about a week ago, so I assume it is the latest version that you've mentioned. To check it, I've typed the following to the CLI and got the following output:

(tomopy) rnice8-0307:Kutsal/ % tomopy --version
astropy module not found
tomopy 0.2

About the example for the error, I should clarify my first comment above further: In CLI, I typed the following:

tomopy find_center --file_name=tomo_00056.h5

where, 'tomo_00056.h5' is an example dataset got from the tomobank webpage. Then, I got the following output on the terminal:

astropy module not found
2021-11-04 13:54:31,531 - Saving log at /home/esrf/kutsal/logs/tomopy_2021-11-04_13_54_31.log
2021-11-04 13:54:31,531 - find center start
2021-11-04 13:54:31,532 - Found: ['tomo_00056.h5']
2021-11-04 13:54:31,532 - Determining the rotation axis location
2021-11-04 13:54:31,532 - *** calculating automatic center
2021-11-04 13:54:31,533 - *** auto pixel size reading
2021-11-04 13:54:31,533 - *** *** OFF
2021-11-04 13:54:31,534 - *** auto reading filter configuration
2021-11-04 13:54:31,534 - *** *** standard
2021-11-04 13:54:31,534 - *** *** skip finding exposure ratio
2021-11-04 13:54:31,534 - *** loading a stardard data set: tomo_00056.h5
2021-11-04 13:54:31,557 - *** tomo_00056.h5 is a valid dx file format
2021-11-04 13:54:31,557 - *** median filter flat images
2021-11-04 13:54:31,558 - *** median filter dark images
2021-11-04 13:54:31,558 - *** correcting for blocked view data collection
2021-11-04 13:54:31,558 - *** *** OFF
2021-11-04 13:54:31,558 - *** binning
2021-11-04 13:54:31,559 - *** *** OFF
2021-11-04 13:54:31,559 - *** rotation center: -1.000000
2021-11-04 13:54:31,559 - *** zinger removal
2021-11-04 13:54:31,559 - *** *** OFF
2021-11-04 13:54:31,559 - *** normalization
*** Error in `/home/esrf/kutsal/anaconda3/envs/tomopy/bin/python': double free or corruption (!prev): 0x000055e341df19e0 ***
Aborted

I get the same error message of "double free or corruption", when I tried to run the reconstruction in CLI with

tomopy recon --file_name=tomo_00056.h5

or if I try to perform the reconstruction through a python interpreter.

[carterbox]

@mkutsal, I was not able to recreate your problem, so I need a bit more information. Could you share the output of the following command with the conda environment that you are using activated:

conda info && conda env export

It will show all the packages installed in your environment, so I can recreate it, and give a bit more info about your system. For example, is rNICE ARM/x86/POWERPC architecture?

[mkutsal]

@carterbox Thanks again for your kind reply. The command in your comment gave the following output:

     active environment : tomopy
    active env location : /home/esrf/kutsal/anaconda3/envs/tomopy
            shell level : 1
       user config file : /home/esrf/kutsal/.condarc
 populated config files : /home/esrf/kutsal/.condarc
          conda version : 4.10.3
    conda-build version : 3.21.4
         python version : 3.8.8.final.0
       virtual packages : __linux=3.16.0=0
                          __glibc=2.19=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/esrf/kutsal/anaconda3  (writable)
      conda av data dir : /home/esrf/kutsal/anaconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/esrf/kutsal/anaconda3/pkgs
                          /home/esrf/kutsal/.conda/pkgs
       envs directories : /home/esrf/kutsal/anaconda3/envs
                          /home/esrf/kutsal/.conda/envs
               platform : linux-64
             user-agent : conda/4.10.3 requests/2.26.0 CPython/3.8.8 Linux/3.16.0-10-amd64 debian/8.11 glibc/2.19
                UID:GID : 200720:20000
             netrc file : None
           offline mode : False

name: tomopy
channels:
  - defaults
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_llvm
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
  - blosc=1.21.0=h9c3ff4c_0
  - brotli=1.0.9=h7f98852_6
  - brotli-bin=1.0.9=h7f98852_6
  - brunsli=0.1=h9c3ff4c_0
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f98852_0
  - c-blosc2=2.0.4=h5f21a17_1
  - ca-certificates=2021.10.8=ha878542_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - certifi=2021.10.8=py39hf3d152e_1
  - cfitsio=4.0.0=h9a35b8e_0
  - cftime=1.5.1.1=py39hce5d2b2_1
  - charls=2.2.0=h9c3ff4c_0
  - cloudpickle=2.0.0=pyhd8ed1ab_0
  - cudatoolkit=11.5.0=h36ae40a_9
  - curl=7.79.1=h494985f_1
  - cycler=0.11.0=pyhd8ed1ab_0
  - cytoolz=0.11.2=py39h3811e60_1
  - dask-core=2021.11.0=pyhd8ed1ab_0
  - debugpy=1.5.1=py39he80948d_0
  - decorator=5.1.0=pyhd8ed1ab_0
  - dxchange=0.1.6=pyhd3deb0d_0
  - dxfile=0.5=py_1
  - edffile=5.0.0=pyh24bf2e0_1000
  - entrypoints=0.3=pyhd8ed1ab_1003
  - freetype=2.10.4=h0708190_1
  - fsspec=2021.11.0=pyhd8ed1ab_0
  - giflib=5.2.1=h36c2ea0_2
  - h5py=3.4.0=nompi_py39h7e08c79_101
  - hdf4=4.2.15=h10796ff_3
  - hdf5=1.12.1=nompi_h7f166f4_101
  - imagecodecs=2021.8.26=py39h571908b_2
  - imageio=2.9.0=py_0
  - ipykernel=6.4.2=py39hef51801_0
  - ipython=7.29.0=py39hef51801_1
  - ipython_genutils=0.2.0=py_1
  - jbig=2.1=h7f98852_2003
  - jedi=0.18.0=py39hf3d152e_3
  - jpeg=9d=h36c2ea0_0
  - jupyter_client=7.0.6=pyhd8ed1ab_0
  - jupyter_core=4.9.1=py39hf3d152e_0
  - jxrlib=1.1=h7f98852_2
  - kiwisolver=1.3.2=py39h1a9c180_1
  - krb5=1.19.2=h48eae69_3
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=3.0=h9c3ff4c_0
  - libaec=1.0.6=h9c3ff4c_0
  - libblas=3.9.0=12_linux64_openblas
  - libbrotlicommon=1.0.9=h7f98852_6
  - libbrotlidec=1.0.9=h7f98852_6
  - libbrotlienc=1.0.9=h7f98852_6
  - libcblas=3.9.0=12_linux64_openblas
  - libcurl=7.79.1=h494985f_1
  - libdeflate=1.8=h7f98852_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libffi=3.4.2=h9c3ff4c_4
  - libgcc-ng=11.2.0=h1d223b6_11
  - libgfortran-ng=11.2.0=h69a702a_11
  - libgfortran5=11.2.0=h5c6108e_11
  - liblapack=3.9.0=12_linux64_openblas
  - libnetcdf=4.8.1=nompi_hb3fd0d9_101
  - libnghttp2=1.43.0=ha19adfc_1
  - libopenblas=0.3.18=pthreads_h8fe5266_0
  - libpng=1.6.37=h21135ba_2
  - libsodium=1.0.18=h36c2ea0_1
  - libssh2=1.10.0=ha35d2d1_2
  - libstdcxx-ng=11.2.0=he4da1e4_11
  - libtiff=4.3.0=h6f004c6_2
  - libwebp-base=1.2.1=h7f98852_0
  - libzip=1.8.0=h1c5bbd1_1
  - libzlib=1.2.11=h36c2ea0_1013
  - libzopfli=1.0.3=h9c3ff4c_0
  - llvm-openmp=12.0.1=h4bd325d_1
  - locket=0.2.0=py_2
  - lz4-c=1.9.3=h9c3ff4c_1
  - matplotlib-base=3.4.3=py39h2fa2bec_1
  - matplotlib-inline=0.1.3=pyhd8ed1ab_0
  - mkl=2020.4=h726a3e6_304
  - mkl_fft=1.3.0=py39hce5d2b2_1
  - ncurses=6.2=h58526e2_4
  - nest-asyncio=1.5.1=pyhd8ed1ab_0
  - netcdf4=1.5.8=nompi_py39h64b754b_101
  - networkx=2.6.3=pyhd8ed1ab_1
  - numexpr=2.7.3=py39hde0f152_1
  - numpy=1.21.4=py39hdbf815f_0
  - olefile=0.46=pyh9f0ad1d_1
  - openjpeg=2.4.0=hb52868f_1
  - openssl=3.0.0=h7f98852_2
  - packaging=21.0=pyhd8ed1ab_0
  - pandas=1.3.4=py39hde0f152_1
  - parso=0.8.2=pyhd8ed1ab_0
  - partd=1.2.0=pyhd8ed1ab_0
  - pexpect=4.8.0=pyh9f0ad1d_2
  - pickleshare=0.7.5=py_1003
  - pillow=8.3.2=py39ha612740_0
  - pip=21.3.1=pyhd8ed1ab_0
  - prompt-toolkit=3.0.22=pyha770c72_0
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pygments=2.10.0=pyhd8ed1ab_0
  - pyparsing=3.0.5=pyhd8ed1ab_0
  - python=3.9.7=hf930737_3_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.9=2_cp39
  - pytz=2021.3=pyhd8ed1ab_0
  - pywavelets=1.1.1=py39hce5d2b2_4
  - pyyaml=6.0=py39h3811e60_3
  - pyzmq=22.3.0=py39h37b5a0c_1
  - readline=8.1=h46c0cb4_0
  - scikit-image=0.17.2=py39hde0f152_4
  - scipy=1.7.1=py39hee8e79c_0
  - setuptools=58.5.3=py39hf3d152e_0
  - six=1.16.0=pyh6c4a22f_0
  - snappy=1.1.8=he1b5a44_3
  - spefile=1.6.1=pyhd3deb0d_0
  - sqlite=3.36.0=h9cd32fc_2
  - tifffile=2021.11.2=pyhd8ed1ab_0
  - tk=8.6.11=h27826a3_1
  - tomopy=1.10.1=cuda11.2py39he25528a_0
  - toolz=0.11.2=pyhd8ed1ab_0
  - tornado=6.1=py39h3811e60_2
  - traitlets=5.1.1=pyhd8ed1ab_0
  - tzdata=2021e=he74cb21_0
  - wcwidth=0.2.5=pyh9f0ad1d_2
  - wheel=0.37.0=pyhd8ed1ab_1
  - xz=5.2.5=h516909a_1
  - yaml=0.2.5=h516909a_0
  - zeromq=4.3.4=h9c3ff4c_1
  - zfp=0.5.5=h9c3ff4c_7
  - zlib=1.2.11=h36c2ea0_1013
  - zstd=1.5.0=ha95c52a_0
  - pip:
    - tomopy-cli==0.3
prefix: /home/esrf/kutsal/anaconda3/envs/tomopy

Sorry for spamming with this long quote. Please let me know if you need any further information.

[carterbox]

I was still not able to recreate the problem, but I didn't exactly recreate your environment because I have cudatoolkit=11.2. Are you using CUDA? Conda says it doesn't detect CUDA, but you have cudatoolkit=11.5 installed. Please try installing tomopy=*=*100. This will install the non-CUDA version of tomopy.

Finally, if none of those suggestions uncover the problem help, my final suggestion is to build TomoPy from source.