Crash in step 6

[lpagani91]

Though I could run the demo version of gausspy+-0.2dev0 under py3.8.7, scipy1.6.0, numpy1.19.5 and lmfit1.0.1, on a Mac OS10.12.6 (Sierra), when I applied it to my own dataset (13CO 1-0 spectra of a single cloud), it crashed in step 6.
It is difficult to report the problem because the iterations are written upward on the screen, erasing all the initial information.
I kill the job before the iterations start to get the initial information but the end is more problematic to collect.

OTF>python step_6_13co_10_spatial_refit2--grs.py

===========================
Spatial refitting - Phase 2

Flagging:

• Blended components: True
• Negative residual features: True
• Broad components: True
  flagged if FWHM of broadest component in spectrum is:

  = 2.0 times the FWHM of second broadest component
  or
  = 2.0 times any FWHM in >= 50% of its neigbors

• High reduced chi2 values (> None): False
• Non-Gaussian distributed residuals: True
• Differing number of components: True

threshold for required components: 0.833

Flags:

• 759 spectra w/ blended components
• 92 spectra w/ negative residual feature
• 1434 spectra w/ broad feature
  (info: 16 spectra w/ a FWHM > 57 channels)
• 0 spectra w/ high rchi2 value
• 1081 spectra w/ residual not passing normality test
• 121 spectra w/ differing number of components

check which spectra require refitting...

determine neighbors for all spectra...
12537it [00:10, 1165.48it/s]

start refit iteration #1...
Using 6 of 8 cpus
6%|âââ | 721/12.5k [00:00<00:07, 1.51kit/s]

then it starts writing upwards... (not very convenient !):
[...]
46%|ââââââââââââââââââ | 5.72k/12.5k [03:02<05:06, 22.2it/s]
43%|âââââââââââââââââ | 5.42k/12.5k [02:47<03:03, 38.8it/s]
43%|âââââââââââââââââ | 5.40k/12.5k [02:47<04:07, 28.9it/s]
41%|ââââââââââââââââ | 5.09k/12.5k [02:36<02:08, 57.7it/s]
41%|ââââââââââââââââ | 5.08k/12.5k [02:35<01:29, 82.9it/s]
41%|ââââââââââââââââ | 5.05k/12.5k [02:35<02:00, 62.0it/s]
38%|âââââââââââââââ | 4.77k/12.5k [02:27<01:44, 74.3it/s]
38%|âââââââââââââââ | 4.76k/12.5k [02:27<01:57, 66.3it/s]
[...]
and finally :

100%|ââââââââââââââââââââââââââââââââââââââ| 12.5k/12.5k [07:06<00:00, 29.4it/s]
12534it [00:00, 427489.74it/s] ââââââââââââ| 12.1k/12.5k [07:02<00:00, 1.08kit/s]
SUCCESS ââââââââââââââââââââââââââââââââââ | 11.3k/12.5k [07:01<00:03, 331it/s]

it seems to have finished with these iterations. The output is mixed with the iteration report and in the end we get (I boldface the useful part, to separate from the iteration leftovers):

check which spectra require refitting... | 8.71k/12.5k [06:19<03:41, 17.3it/s]
Traceback (most recent call last): | 8.70k/12.5k [06:18<02:47, 22.9it/s]
File "step_6_13co_10_spatial_refit2--grs.py", line 86, in 3, 24.9it/s]
main()
File "step_6_13co_10_spatial_refit2--grs.py", line 53, in main
sp.spatial_fitting(continuity=True)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/gausspyplus-0.2.dev0-py3.8.egg/gausspyplus/spatial_fitting.py", line 323, in spatial_fitting
self.check_continuity()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/gausspyplus-0.2.dev0-py3.8.egg/gausspyplus/spatial_fitting.py", line 2633, in check_continuity
self.refitting()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/gausspyplus-0.2.dev0-py3.8.egg/gausspyplus/spatial_fitting.py", line 886, in refitting
self.check_continuity()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/gausspyplus-0.2.dev0-py3.8.egg/gausspyplus/spatial_fitting.py", line 2632, in check_continuity
self.check_indices_refit()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/gausspyplus-0.2.dev0-py3.8.egg/gausspyplus/spatial_fitting.py", line 2615, in check_indices_refit
self.indices_refit = np.delete(self.indices_all.copy(), indices_remove)
File "<array_function internals>", line 5, in delete
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/lib/function_base.py", line 4406, in delete
keep[obj,] = False
IndexError: index 12568 is out of bounds for axis 0 with size 12537

so its index goes too far by a relatively large amount...

[lpagani91]

I entered the wrong input file (*fit_fin.pickle) instead of (*fit_fin_sf-p1.pickle) but it still crashes in a similar way:

File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/lib/function_base.py", line 4406, in delete
keep[obj,] = False
IndexError: index 12541 is out of bounds for axis 0 with size 12537

[lpagani91]

The problem seems absent on a Linux machine:

Linux 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64

with python 3.7, and scipy 1.4.1

but the printing upwards of the iterations is still present and is painful. If this could be cured, that would be great.

[mriener]

Not sure, but based on your last findings it seems indeed like newer versions of scipy (> 1.4.1) might cause issues (also confirming issue #2).

However, have I understood it correctly that you managed to run the full GRS tutorial (Tutorial_example-GRS.ipynb) without any crashes with Python 3.8.7, scipy 1.6.0, numpy 1.19.5, and lmfit 1.0.1? If yes, then the crash in phase 2 of the spatial refitting (step 6) that you report for your own data might be more indicative that something else went wrong here.

Concerning the upward printing of the progress bar: this seems to be a known issue with tqdm.

Are you by any chance running GaussPy+ on a machine that you are connected with via ssh and screen? Your reported problems with the progress bar seem very similar to the issue reported here on Stack Overflow.

The information printed to the terminal is by default also logged in the directory decompose/gpy_log, so you can always retrieve it from there, in case the upward printed progress bar erased it in the terminal.

[lpagani91]

Not sure, but based on your last findings it seems indeed like newer versions of scipy (> 1.4.1) might cause issues (also confirming issue #2 <#2>).

No. See below

However, have I understood it correctly that you managed to run the full GRS tutorial (Tutorial_example-GRS.ipynb) without any crashes with Python 3.8.7, scipy 1.6.0, numpy 1.19.5, and lmfit 1.0.1? If yes, then the crash in phase 2 of the spatial refitting (step 6) that you report for your own data might be more indicative that something else went wrong here.

both tutorials were run on my Mac OS 10.12.6, etc. without any trouble indeed.

Concerning the upward printing of the progress bar: this seems to be a known issue with tqdm <tqdm/tqdm#798>.

I see. Too bad.

Are you by any chance running GaussPy+ on a machine that you are connected with via ssh and screen? Your reported problems with the progress bar seem very similar to the issue reported here on Stack Overflow <https://stackoverflow.com/questions/52250556/tqdm-in-screen-environment-printing-new-line-and-unknown-charactors>.

The problem is the same on my local MacBookPro and on the remote Linux machine for which I am connected with via ssh. I have no idea about screen. I just type ssh -Y in one of my xterm windows locally.

The information printed to the terminal is by default also logged in the directory decompose/gpy_log, so you can always retrieve it from there, in case the upward printed progress bar erased it in the terminal.

except that there is no log written for step_6 even in the Linux machine where it did not crash, starting from step 3 (because the training was already done before) : 20210212-130534_g+_preparation_L134_13co_10_vertical.log 20210212-130548_g+_decomposition_L134_13co_10_vertical.log 20210212-130654_g+_spatial_refitting_L134_13co_10_vertical_g+_fit_fin.log 20210212-130806_g+_spatial_refitting_L134_13co_10_vertical_g+_fit_fin_sf-p1.log and that’s it! Laurent

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADJVTNBMBMOV7YTNBAVRAHLS6VQXDANCNFSM4XOSRGEQ>.

"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème" (devise Shadok)