Error running fmriprep on HCP data

[DanielDiGiovanni-py]

What happened?

I got the following error, not sure how to interpret it:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/bin/fmriprep", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/fmriprep/cli/run.py", line 102, in main
    retval = mgr.dict()
  File "/opt/conda/envs/fmriprep/lib/python3.10/multiprocessing/managers.py", line 723, in temp
    token, exp = self._create(typeid, *args, **kwds)
  File "/opt/conda/envs/fmriprep/lib/python3.10/multiprocessing/managers.py", line 606, in _create
    conn = self._Client(self._address, authkey=self._authkey)
  File "/opt/conda/envs/fmriprep/lib/python3.10/multiprocessing/connection.py", line 502, in Client
    c = SocketClient(address)
  File "/opt/conda/envs/fmriprep/lib/python3.10/multiprocessing/connection.py", line 630, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory
Exception ignored in atexit callback: <function AtexitIntegration.setup_once.<locals>._shutdown at 0x7f914907b370>
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sentry_sdk/integrations/atexit.py", line 61, in _shutdown
    client.close(callback=integration.callback)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sentry_sdk/client.py", line 572, in close
    self.flush(timeout=timeout, callback=callback)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sentry_sdk/client.py", line 594, in flush
    self.transport.flush(timeout=timeout, callback=callback)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sentry_sdk/transport.py", line 492, in flush
    self._worker.submit(lambda: self._flush_client_reports(force=True))
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sentry_sdk/worker.py", line 113, in submit
    self._ensure_thread()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sentry_sdk/worker.py", line 42, in _ensure_thread
    self.start()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sentry_sdk/worker.py", line 70, in start
    self._thread.start()
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/sentry_sdk/integrations/threading.py", line 56, in sentry_start
    return old_start(self, *a, **kw)
  File "/opt/conda/envs/fmriprep/lib/python3.10/threading.py", line 935, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

What command did you use?

fmriprep-docker /data/data02/ddigio/data_bids/ /data/data02/ddigio/output/ participant --skip_bids_validation

What version of fMRIPrep are you running?

23.1.4

How are you running fMRIPrep?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

No response

Additional information / screenshots

I'm skipping BIDS validation, because I would run out of memory when I included it.

[effigies]

This appears to be an issue with spawning subprocesses. What do you get when you run ulimit -a on the system that this is supposed to run on? How much available memory does the system have?

[DanielDiGiovanni-py]

The system has 64Mb of memory, and it should all be accessible to the process

[DanielDiGiovanni-py]

If it's a problem spawning subprocesses maybe one of these arguments would help with this? --nprocs, --nthreads, --n_cpus, --n-cpus, --omp-nthreads

[effigies]

No. Unfortunately, if you can't spawn subprocesses, you're not going to be able to run the tool.

[effigies]

It's possible. That running BIDS validation hits memory issues would make me think that you don't actually have access to all of it.

If there's not a ulimit issue preventing you from starting subprocesses, it's unclear what could be causing this. You could try reproducing with:

$ fmriprep-docker /data/data02/ddigio/data_bids/ /data/data02/ddigio/output/ participant --shell
> python
>>> from multiprocessing import Manager
>>> with Manager as mgr:
...     retval = mgr.dict()

Assuming that reproduces, I can't really think what to do with that. I suppose open an issue on cpython.