Randomly fail when using functions in anat_fit_wf #3277
Labels
Comments
|
The only thing I can find about "upt: Subscript out of range." is freesurfer/freesurfer#1042, but |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
We are running fmriprep within docker for our own subject database (rs-fMRI, T1, T2). The laptop I'm using has M3 chip, 32GB memory, and 12 cores. Some of the subjects pass successfully, but the rest of them failed on different types of nodes:
I tried running fmriprep on only one subject (subject 14) with two sessions, it failed on a different error.
Another attempt fmriprep on only subject 15 with two sessions is successful. Since it failed at the first time, I think it's similar to issue #3220:#3220 (comment), but our data is located at Desktop directory of Mac which I believe is not FAT/NTFS.
What command did you use?
What version of fMRIPrep are you running?
23.2.1
How are you running fMRIPrep?
Docker
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
No
Please copy and paste any relevant log output.
This is the log from the attempt with subject 14 only: Node Name: fmriprep_23_2_wf.sub_14_wf.anat_fit_wf.surface_recon_wf.autorecon1 File: /out/sub-14/log/20240430-190529_47e10b7b-b8e4-470a-b82e-8aa193ebf15d/crash-20240430-191720-root-autorecon1-cb641879-36eb-42b0-9e06-aca5f6b0a846.txt Working Directory: /scratch/fmriprep_23_2_wf/sub_14_wf/anat_fit_wf/surface_recon_wf/autorecon1 Inputs: FLAIR_file: T1_files: T2_file: args: big_ventricles: brainstem: directive: autorecon1 environ: {} expert: flags: hemi: hippocampal_subfields_T1: hippocampal_subfields_T2: hires: mprage: mri_aparc2aseg: mri_ca_label: mri_ca_normalize: mri_ca_register: mri_edit_wm_with_aseg: mri_em_register: mri_fill: mri_mask: mri_normalize: mri_pretess: mri_remove_neck: mri_segment: mri_segstats: mri_tessellate: mri_watershed: mris_anatomical_stats: mris_ca_label: mris_fix_topology: mris_inflate: mris_make_surfaces: mris_register: mris_smooth: mris_sphere: mris_surf2vol: mrisp_paint: openmp: 6 parallel: steps: subject_id: recon_all subjects_dir: talairach: use_FLAIR: use_T2: xopts: Traceback (most recent call last): File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run result = self._run_interface(execute=True) File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface return self._run_command(execute) File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command raise NodeExecutionError(msg) nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node autorecon1. Cmdline: recon-all -autorecon1 -i /scratch/fmriprep_23_2_wf/sub_14_wf/anat_fit_wf/anat_template_wf/anat_merge/sub-14_ses-01_T1w_noise_corrected_ras_template.nii -T2 /data/sub-14/ses-01/anat/sub-14_ses-01_T2w.nii -noskullstrip -noT2pial -noFLAIRpial -openmp 6 -subjid sub-14 -sd /out/sourcedata/freesurfer Stdout: fs-check-version --s sub-14 --o /tmp/tmp.OIwy39 Tue Apr 30 19:11:07 UTC 2024 setenv SUBJECTS_DIR /out/sourcedata/freesurfer cd /scratch/fmriprep_23_2_wf/sub_14_wf/anat_fit_wf/surface_recon_wf/autorecon1 /opt/freesurfer/bin/fs-check-version --s sub-14 --o /tmp/tmp.OIwy39 -rwxrwxr-x 1 root root 18565 Aug 4 2022 /opt/freesurfer/bin/fs-check-version freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 $Id$ Linux 005abe4ae096 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux pid 468 Current FS Version freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 Subject does not have a bstampfile, copying /opt/freesurfer/build-stamp.txt Subject FS Version: freesurfer-linux-ubuntu22_x86_64-7.3.2-20220804-6354275 No constraints on version because REQ=UnSet and FsVerFile=NotThere #@#% fs-check-version match = 1 fs-check-version Done INFO: SUBJECTS_DIR is /out/sourcedata/freesurfer Actual FREESURFER_HOME /opt/freesurfer Linux 005abe4ae096 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux /out/sourcedata/freesurfer/sub-14 mri_convert /scratch/fmriprep_23_2_wf/sub_14_wf/anat_fit_wf/anat_template_wf/anat_merge/sub-14_ses-01_T1w_noise_corrected_ras_template.nii /out/sourcedata/freesurfer/sub-14/mri/orig/001.mgz mri_convert /scratch/fmriprep_23_2_wf/sub_14_wf/anat_fit_wf/anat_template_wf/anat_merge/sub-14_ses-01_T1w_noise_corrected_ras_template.nii /out/sourcedata/freesurfer/sub-14/mri/orig/001.mgz reading from /scratch/fmriprep_23_2_wf/sub_14_wf/anat_fit_wf/anat_template_wf/anat_merge/sub-14_ses-01_T1w_noise_corrected_ras_template.nii... TR=0.00, TE=0.00, TI=0.00, flip angle=0.00 i_ras = (0.998126, 0.0459843, -0.0403783) j_ras = (-0.0317161, 0.952999, 0.301308) k_ras = (0.052336, -0.299462, 0.952672) writing to /out/sourcedata/freesurfer/sub-14/mri/orig/001.mgz... #-------------------------------------------- #@# T2/FLAIR Input Tue Apr 30 19:11:17 UTC 2024 /out/sourcedata/freesurfer/sub-14 mri_convert --no_scale 1 /data/sub-14/ses-01/anat/sub-14_ses-01_T2w.nii /out/sourcedata/freesurfer/sub-14/mri/orig/T2raw.mgz mri_convert --no_scale 1 /data/sub-14/ses-01/anat/sub-14_ses-01_T2w.nii /out/sourcedata/freesurfer/sub-14/mri/orig/T2raw.mgz reading from /data/sub-14/ses-01/anat/sub-14_ses-01_T2w.nii... TR=4800.00, TE=0.00, TI=0.00, flip angle=0.00 i_ras = (0.0317161, -0.952999, -0.301308) j_ras = (0.052336, -0.299462, 0.952672) k_ras = (0.998126, 0.0459843, -0.0403783) writing to /out/sourcedata/freesurfer/sub-14/mri/orig/T2raw.mgz... #-------------------------------------------- #@# MotionCor Tue Apr 30 19:11:34 UTC 2024 Found 1 runs /out/sourcedata/freesurfer/sub-14/mri/orig/001.mgz Checking for (invalid) multi-frame inputs... Only one run found so motion correction will not be performed. I'll copy the run to rawavg and continue. cp /out/sourcedata/freesurfer/sub-14/mri/orig/001.mgz /out/sourcedata/freesurfer/sub-14/mri/rawavg.mgz mri_info /out/sourcedata/freesurfer/sub-14/mri/rawavg.mgz rawavg.mgz ======================================== Volume information for /out/sourcedata/freesurfer/sub-14/mri/rawavg.mgz type: MGH dimensions: 224 x 220 x 224 voxel sizes: 1.000000, 1.000000, 1.000000 type: FLOAT (3) fov: 224.000 dof: 1 xstart: -112.0, xend: 112.0 ystart: -110.0, yend: 110.0 zstart: -112.0, zend: 112.0 TR: 0.00 msec, TE: 0.00 msec, TI: 0.00 msec, flip angle: 0.00 degrees nframes: 1 PhEncDir: UNKNOWN FieldStrength: 0.000000 ras xform present xform info: x_r = 0.9981, y_r = -0.0317, z_r = 0.0523, c_r = 2.7663 : x_a = 0.0460, y_a = 0.9530, z_a = -0.2995, c_a = -11.0132 : x_s = -0.0404, y_s = 0.3013, z_s = 0.9527, c_s = -5.3312 talairach xfm : Orientation : RAS Primary Slice Direction: axial voxel to ras transform: 0.9981 -0.0317 0.0523 -111.3966 0.0460 0.9530 -0.2995 -87.4536 -0.0404 0.3013 0.9527 -140.6519 0.0000 0.0000 0.0000 1.0000 voxel-to-ras determinant 1 ras to voxel transform: 0.9981 0.0460 -0.0404 109.5300 -0.0317 0.9530 0.3013 122.1897 0.0523 -0.2995 0.9527 113.6361 0.0000 0.0000 0.0000 1.0000 /out/sourcedata/freesurfer/sub-14 mri_convert /out/sourcedata/freesurfer/sub-14/mri/rawavg.mgz /out/sourcedata/freesurfer/sub-14/mri/orig.mgz --conform mri_convert /out/sourcedata/freesurfer/sub-14/mri/rawavg.mgz /out/sourcedata/freesurfer/sub-14/mri/orig.mgz --conform reading from /out/sourcedata/freesurfer/sub-14/mri/rawavg.mgz... TR=0.00, TE=0.00, TI=0.00, flip angle=0.00 i_ras = (0.998126, 0.0459843, -0.0403783) j_ras = (-0.0317161, 0.952999, 0.301308) k_ras = (0.052336, -0.299462, 0.952672) changing data type from float to uchar (noscale = 0)... MRIchangeType: Building histogram -12222.6 240095 1000, flo=0, fhi=0.999, dest_type=0 Reslicing using trilinear interpolation writing to /out/sourcedata/freesurfer/sub-14/mri/orig.mgz... mri_add_xform_to_header -c /out/sourcedata/freesurfer/sub-14/mri/transforms/talairach.xfm /out/sourcedata/freesurfer/sub-14/mri/orig.mgz /out/sourcedata/freesurfer/sub-14/mri/orig.mgz INFO: extension is mgz mri_info /out/sourcedata/freesurfer/sub-14/mri/orig.mgz orig.mgz ======================================== Volume information for /out/sourcedata/freesurfer/sub-14/mri/orig.mgz type: MGH dimensions: 256 x 256 x 256 voxel sizes: 1.000000, 1.000000, 1.000000 type: UCHAR (0) fov: 256.000 dof: 1 xstart: -128.0, xend: 128.0 ystart: -128.0, yend: 128.0 zstart: -128.0, zend: 128.0 TR: 0.00 msec, TE: 0.00 msec, TI: 0.00 msec, flip angle: 0.00 degrees nframes: 1 PhEncDir: UNKNOWN FieldStrength: 0.000000 ras xform present xform info: x_r = -1.0000, y_r = 0.0000, z_r = 0.0000, c_r = 2.7663 : x_a = 0.0000, y_a = 0.0000, z_a = 1.0000, c_a = -11.0132 : x_s = 0.0000, y_s = -1.0000, z_s = -0.0000, c_s = -5.3312 talairach xfm : /out/sourcedata/freesurfer/sub-14/mri/transforms/talairach.xfm Orientation : LIA Primary Slice Direction: coronal voxel to ras transform: -1.0000 0.0000 0.0000 130.7663 0.0000 0.0000 1.0000 -139.0132 0.0000 -1.0000 -0.0000 122.6688 0.0000 0.0000 0.0000 1.0000 voxel-to-ras determinant -1 ras to voxel transform: -1.0000 -0.0000 -0.0000 130.7663 -0.0000 -0.0000 -1.0000 122.6688 0.0000 1.0000 -0.0000 139.0132 -0.0000 -0.0000 -0.0000 1.0000 #-------------------------------------------- #@# Talairach Tue Apr 30 19:11:42 UTC 2024 /out/sourcedata/freesurfer/sub-14/mri mri_nu_correct.mni --no-rescale --i orig.mgz --o orig_nu.mgz --ants-n4 --n 1 --proto-iters 1000 --distance 50 /usr/bin/bc /out/sourcedata/freesurfer/sub-14/mri /opt/freesurfer/bin/mri_nu_correct.mni --no-rescale --i orig.mgz --o orig_nu.mgz --ants-n4 --n 1 --proto-iters 1000 --distance 50 nIters 1 mri_nu_correct.mni 7.3.2 Linux 005abe4ae096 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Tue Apr 30 19:11:43 UTC 2024 tmpdir is ./tmp.mri_nu_correct.mni.835 cd /out/sourcedata/freesurfer/sub-14/mri AntsN4BiasFieldCorrectionFs -i orig.mgz -o ./tmp.mri_nu_correct.mni.835/nu0.mgz --dtype uchar AntsN4BiasFieldCorrectionFs done mri_convert ./tmp.mri_nu_correct.mni.835/nu0.mgz orig_nu.mgz --like orig.mgz --conform mri_convert ./tmp.mri_nu_correct.mni.835/nu0.mgz orig_nu.mgz --like orig.mgz --conform reading from ./tmp.mri_nu_correct.mni.835/nu0.mgz... TR=0.00, TE=0.00, TI=0.00, flip angle=0.00 i_ras = (-1, 1.86265e-09, 0) j_ras = (0, 0, -1) k_ras = (0, 1, -2.98023e-08) INFO: transform src into the like-volume: orig.mgz writing to orig_nu.mgz... Tue Apr 30 19:14:07 UTC 2024 mri_nu_correct.mni done talairach_avi --i orig_nu.mgz --xfm transforms/talairach.auto.xfm talairach_avi log file is transforms/talairach_avi.log... mv -f /out/sourcedata/freesurfer/sub-14/mri/talsrcimg_to_711-2C_as_mni_average_305_t4_vox2vox.txt /out/sourcedata/freesurfer/sub-14/mri/transforms/talsrcimg_to_711-2C_as_mni_average_305_t4_vox2vox.txt Started at Tue Apr 30 19:14:08 UTC 2024 Ended at Tue Apr 30 19:14:29 UTC 2024 talairach_avi done cp transforms/talairach.auto.xfm transforms/talairach.xfm lta_convert --src orig.mgz --trg /opt/freesurfer/average/mni305.cor.mgz --inxfm transforms/talairach.xfm --outlta transforms/talairach.xfm.lta --subject fsaverage --ltavox2vox 7.3.2 --src: orig.mgz src image (geometry). --trg: /opt/freesurfer/average/mni305.cor.mgz trg image (geometry). --inmni: transforms/talairach.xfm input MNI/XFM transform. --outlta: transforms/talairach.xfm.lta output LTA. --s: fsaverage subject name --ltavox2vox: output LTA as VOX_TO_VOX transform. LTA read, type : 1 0.98559 0.03697 -0.04689 -0.52249; -0.02291 1.06278 0.24999 -3.72581; 0.04425 -0.25222 0.99025 -0.46518; 0.00000 0.00000 0.00000 1.00000; setting subject to fsaverage Writing LTA to file transforms/talairach.xfm.lta... lta_convert successful. /out/sourcedata/freesurfer/sub-14/mri/transforms /out/sourcedata/freesurfer/sub-14/mri /out/sourcedata/freesurfer/sub-14/mri #-------------------------------------------- #@# Talairach Failure Detection Tue Apr 30 19:14:31 UTC 2024 /out/sourcedata/freesurfer/sub-14/mri talairach_afd -T 0.005 -xfm transforms/talairach.xfm talairach_afd: Talairach Transform: transforms/talairach.xfm OK (p=0.6727, pval=0.3663 >= threshold=0.0050) awk -f /opt/freesurfer/bin/extract_talairach_avi_QA.awk /out/sourcedata/freesurfer/sub-14/mri/transforms/talairach_avi.log tal_QC_AZS /out/sourcedata/freesurfer/sub-14/mri/transforms/talairach_avi.log TalAviQA: 0.97662 z-score: 0 #-------------------------------------------- #@# Nu Intensity Correction Tue Apr 30 19:14:31 UTC 2024 mri_nu_correct.mni --i orig.mgz --o nu.mgz --uchar transforms/talairach.xfm --n 2 --ants-n4 /usr/bin/bc /out/sourcedata/freesurfer/sub-14/mri /opt/freesurfer/bin/mri_nu_correct.mni --i orig.mgz --o nu.mgz --uchar transforms/talairach.xfm --n 2 --ants-n4 nIters 2 mri_nu_correct.mni 7.3.2 Linux 005abe4ae096 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux Tue Apr 30 19:14:32 UTC 2024 tmpdir is ./tmp.mri_nu_correct.mni.1094 cd /out/sourcedata/freesurfer/sub-14/mri AntsN4BiasFieldCorrectionFs -i orig.mgz -o ./tmp.mri_nu_correct.mni.1094/nu0.mgz --dtype uchar AntsN4BiasFieldCorrectionFs done mri_binarize --i ./tmp.mri_nu_correct.mni.1094/nu0.mgz --min -1 --o ./tmp.mri_nu_correct.mni.1094/ones.mgz 7.3.2 cwd /out/sourcedata/freesurfer/sub-14/mri cmdline mri_binarize --i ./tmp.mri_nu_correct.mni.1094/nu0.mgz --min -1 --o ./tmp.mri_nu_correct.mni.1094/ones.mgz sysname Linux hostname 005abe4ae096 machine x86_64 user root input ./tmp.mri_nu_correct.mni.1094/nu0.mgz frame 0 nErode3d 0 nErode2d 0 output ./tmp.mri_nu_correct.mni.1094/ones.mgz Binarizing based on threshold min -1 max +infinity binval 1 binvalnot 0 fstart = 0, fend = 0, nframes = 1 Starting parallel 1 Found 16777216 values in range Counting number of voxels in first frame Found 16777215 voxels in final mask Writing output to ./tmp.mri_nu_correct.mni.1094/ones.mgz Count: 16777215 16777215.000000 16777216 99.999994 mri_binarize done mri_segstats --id 1 --seg ./tmp.mri_nu_correct.mni.1094/ones.mgz --i orig.mgz --sum ./tmp.mri_nu_correct.mni.1094/sum.junk --avgwf ./tmp.mri_nu_correct.mni.1094/input.mean.dat 7.3.2 cwd cmdline mri_segstats --id 1 --seg ./tmp.mri_nu_correct.mni.1094/ones.mgz --i orig.mgz --sum ./tmp.mri_nu_correct.mni.1094/sum.junk --avgwf ./tmp.mri_nu_correct.mni.1094/input.mean.dat sysname Linux hostname 005abe4ae096 machine x86_64 user root whitesurfname white UseRobust 0 Loading ./tmp.mri_nu_correct.mni.1094/ones.mgz Loading orig.mgz Voxel Volume is 1 mm^3 Generating list of segmentation ids Found 1 segmentations Computing statistics for each segmentation Reporting on 1 segmentations Using PrintSegStat Computing spatial average of each frame Writing to ./tmp.mri_nu_correct.mni.1094/input.mean.dat mri_segstats done mri_segstats --id 1 --seg ./tmp.mri_nu_correct.mni.1094/ones.mgz --i ./tmp.mri_nu_correct.mni.1094/nu0.mgz --sum ./tmp.mri_nu_correct.mni.1094/sum.junk --avgwf ./tmp.mri_nu_correct.mni.1094/output.mean.dat 7.3.2 cwd cmdline mri_segstats --id 1 --seg ./tmp.mri_nu_correct.mni.1094/ones.mgz --i ./tmp.mri_nu_correct.mni.1094/nu0.mgz --sum ./tmp.mri_nu_correct.mni.1094/sum.junk --avgwf ./tmp.mri_nu_correct.mni.1094/output.mean.dat sysname Linux hostname 005abe4ae096 machine x86_64 user root whitesurfname white UseRobust 0 Loading ./tmp.mri_nu_correct.mni.1094/ones.mgz Loading ./tmp.mri_nu_correct.mni.1094/nu0.mgz Voxel Volume is 1 mm^3 Generating list of segmentation ids Found 1 segmentations Computing statistics for each segmentation Reporting on 1 segmentations Using PrintSegStat Computing spatial average of each frame Writing to ./tmp.mri_nu_correct.mni.1094/output.mean.dat mri_segstats done mris_calc -o ./tmp.mri_nu_correct.mni.1094/nu0.mgz ./tmp.mri_nu_correct.mni.1094/nu0.mgz mul 1.37393233565938209479 Saving result to './tmp.mri_nu_correct.mni.1094/nu0.mgz' (type = MGH ) [ ok ] mri_convert ./tmp.mri_nu_correct.mni.1094/nu0.mgz nu.mgz --like orig.mgz mri_convert ./tmp.mri_nu_correct.mni.1094/nu0.mgz nu.mgz --like orig.mgz reading from ./tmp.mri_nu_correct.mni.1094/nu0.mgz... TR=0.00, TE=0.00, TI=0.00, flip angle=0.00 i_ras = (-1, 1.86265e-09, 0) j_ras = (0, 0, -1) k_ras = (0, 1, -2.98023e-08) INFO: transform src into the like-volume: orig.mgz writing to nu.mgz... mri_make_uchar nu.mgz transforms/talairach.xfm nu.mgz type change took 0 minutes and 4 seconds. FIRST_PERCENTILE 0.010000 WM_PERCENTILE 0.900000 MAX_R 50.000000 i1 = 10, i2 = 64 #mri_make_uchar# mapping 24 152 to 3 110 : b -17.3481 m 0.838262 : thresh 20.6954 maxsat 324.896 : nzero 12380809 nsat 0 Tue Apr 30 19:17:17 UTC 2024 mri_nu_correct.mni done mri_add_xform_to_header -c /out/sourcedata/freesurfer/sub-14/mri/transforms/talairach.xfm nu.mgz nu.mgz INFO: extension is mgz #-------------------------------------------- #@# Intensity Normalization Tue Apr 30 19:17:18 UTC 2024 /out/sourcedata/freesurfer/sub-14/mri mri_normalize -g 1 -seed 1234 -mprage nu.mgz T1.mgz upt: Subscript out of range. Linux 005abe4ae096 6.6.16-linuxkit #1 SMP Fri Feb 16 11:54:02 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux recon-all -s sub-14 exited with ERRORS at Tue Apr 30 19:17:18 UTC 2024 For more details, see the log file /out/sourcedata/freesurfer/sub-14/scripts/recon-all.log To report a problem, see http://surfer.nmr.mgh.harvard.edu/fswiki/BugReporting Stderr: Traceback: RuntimeError: subprocess exited with code 1.Additional information / screenshots
No response
The text was updated successfully, but these errors were encountered: