fMRI prep error on magnitude1 fieldmap in /extra_data folders despite .bidsignore file including /extra_data folder

[lianneremie]

What happened?

I tried to run fMRI prep, but it failed and gave the following output: fMRIPrep failed: The 'mag_files' trait of a _CheckRegisterInputSpec instance must be a list of from 1 to 2 items which are a pathlike object or string representing an existing file, but a value of ['/project/3033003.02/bids/sub-001/ses-mri02/extra_data/sub-001_ses-mri02_acq-fieldmap_magnitude1.nii.gz', '/project/3033003.02/bids/sub-001/ses-mri02/fmap/sub-001_ses-mri02_acq-fieldmap_magnitude1.nii.gz', '/project/3033003.02/bids/sub-001/ses-mri02/fmap/sub-001_ses-mri02_acq-fieldmap_magnitude2.nii.gz'] <class 'list'> was specified.
indicating that it is not allowed to have more than two magnitude fieldmaps in the bidsfolder per sub-ses.

An extra magnitude1 fieldmap is present in the /extra_data folder as result of a skullstripping step. This /extra_data folder should be ignored, as indicated in de .bidsignore file, but fMRI prep still fails.

Please find the .bidsignore file below:

What command did you use?

# Run step 4. Run preprocessing with FMRIPREP (-f: fmriprep)
        if args.fmriprep is True:
            # If specific session specified create json file to filter bids input for fmriprep
            if sessions:
                # Adapt default fmriprep bids queries to include session numbers and create json file to filter bids input for fmriprep
                query={
                    "fmap": {"datatype": "fmap", "session": sessions},
                    "bold": {"datatype": "func", "suffix": "bold", "session": sessions},
                    "sbref": {"datatype": "func", "suffix": "sbref", "session": sessions},
                    "flair": {"datatype": "anat", "suffix": "FLAIR", "session": sessions},
                    "t2w": {"datatype": "anat", "suffix": "T2w", "session": sessions},
                    "t1w": {"datatype": "anat", "suffix": "T1w", "session": sessions},
                    "roi": {"datatype": "anat", "suffix": "roi", "session": sessions}
                    }
                filterfile = Path(dirs['log']).joinpath("bids_filter_fmriprep.json")
                with filterfile.open('w') as filtfile:
                    json.dump(query,filtfile,indent=4)

            # Build command
            if sessions:
                cmd = f"fmriprep_sub.py {dirs['bids']} -o {os.path.join(dirs['deriv'],'fmriprep')} -m 40000 -p {subject} -a ' --output-spaces MNI152NLin6Asym:res-2 --bids-filter-file {filterfile}' -q ' -e {dirs['log']} -o {dirs['log']}'"
            else:
                cmd = f"fmriprep_sub.py {dirs['bids']} -o {os.path.join(dirs['deriv'],'fmriprep')} -m 40000 -p {subject} -a ' --output-spaces MNI152NLin6Asym:res-2' -q ' -e {dirs['log']} -o {dirs['log']}'"
            
            # Add --force option if requested, otherwise only the newline character, and write the command to the jobfile
            if args.force is True:
                cmd = cmd + " -f\n"
            else:
                cmd = cmd + "\n"
            jobfile.write(cmd)

What version of fMRIPrep are you running?

23.1.0

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Error file:

You are using fMRIPrep-23.1.0, and a newer version of fMRIPrep is available: 23.1.4.
Please check out our documentation about how and when to upgrade:
https://fmriprep.readthedocs.io/en/latest/faq.html#upgrading
(node:29970) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)

Output file (last sentence):

240109-13:22:24,479 nipype.workflow CRITICAL:
	 fMRIPrep failed: The 'mag_files' trait of a _CheckRegisterInputSpec instance must be a list of from 1 to 2 items which are a pathlike object or string representing an existing file, but a value of ['/project/3033003.02/bids/sub-001/ses-mri02/extra_data/sub-001_ses-mri02_acq-fieldmap_magnitude1.nii.gz', '/project/3033003.02/bids/sub-001/ses-mri02/fmap/sub-001_ses-mri02_acq-fieldmap_magnitude1.nii.gz', '/project/3033003.02/bids/sub-001/ses-mri02/fmap/sub-001_ses-mri02_acq-fieldmap_magnitude2.nii.gz'] <class 'list'> was specified.

Additional information / screenshots

No response

[julfou81]

Hi @lianneremie,

If you add the line **/extra_data in your .bidsignore file, does it change the fmriprep behaviour? (I got this idea looking at this page)

Also, could you share the tree of your BIDS folder? Where did you put the .bidsignore file?

[effigies]

Sorry for not responding sooner, and I'm not on my computer so I'm not prepared to look at this in detail. Before you spend more time on it, .bidsignore has no effect on fmriprep. If you have things that you're ignoring that interfere with fmriprep's logic, ignoring them won't help. It'll just stop the validator telling you about them.

[lianneremie]

Sorry for not responding sooner, and I'm not on my computer so I'm not prepared to look at this in detail. Before you spend more time on it, .bidsignore has no effect on fmriprep. If you have things that you're ignoring that interfere with fmriprep's logic, ignoring them won't help. It'll just stop the validator telling you about them.

Thanks for your reply! For now I just work around this issue by moving these files to a different folder in the derivatives. Would there be another way to let fmriprep ignore these files?

[julfou81]

@lianneremie , did you fill your fieldmap json files with IntendedFor to tell which functional images need to be corrected?

I am thinking about this discussion.

But perhaps the error you received is triggered before this filtering happens.

Anyway, I would recommend to have your dataset BIDS compatible if possible to start with, and to always move this folder extra_data somewhere else (for instance in sourcedata/ folder).

[lianneremie]

@julfou81 My dataset is BIDS compatible, my folder looks like this:

The .bidsignore file is directly under the /bids folder, but this is only used for the bidsvalidator at the start of fmriprep as @effigies commented.
The extra_data folders is subject-session specific, so I think it should stay where it is as it also contains other data (like QSM files). Or do you still suggest another location?

I do not use IntendedFor fields (as this gave problems earlier) but I link by functional scans to the fieldmaps using the B0fieldIdentifier and B0fieldSource fields. This might be the problem, as the B0fieldIdentifier tag is still there in the magnitude1 image within the extra_data folder.

[julfou81]

ok, I forgot that extra_data/ was part of the BIDS specification. Indeed, you should not need to remove it then.

How does your bids-filter file looks like?

Yes, maybe removing the B0FieldSource in the json files in the extra_data/ folder will help?

[effigies]

Okay, catching up on all of this:

1. .bidsignore does not make a dataset valid; it hides invalid bits from the validator. This is for the purpose of adding yet-to-be-standardized features to your dataset so that you don't have to contort the dataset by hiding your regular data in sourcedata/ or similar. It is use-at-your-own-risk, though, and surprising behavior of BIDS Apps is one of the risks.
2. I don't understand why scans.tsv or perf/ are being ignored. Both of these are in BIDS, so ignoring them would be concerning if I were reviewing a dataset. On the other hand, fMRIPrep won't do anything with these, so they're fine here.
3. extra_data/ is not part of the BIDS specification, and I would hope that data could be labelled more informatively in a way that is actually queryable (and hence excludable from queries). If they're just additional runs of fieldmaps, then you could still include them in fmap/ but use acq- or run- or similar to distinguish them from other fieldmaps. Your BIDS filter file could include the known good acquisition/run.
4. B0FieldIdentifier is meant to be used to identify scans that are used to create a single fieldmap estimate. If two scans are not meant to be used together, they should have different identifiers.