nf-core download OperationalError: disk I/O error

[bcardus]

Description of the bug

Hello,

I am trying to download epi2me-labs/wf-basecalling to a cluster entry node using nf-core download. I input the following command:

nf-core download --force -l docker.io epi2me-labs/wf-basecalling

and I click through the options to select version etc. However, I keep getting an error culminating in "OperationalError: disk I/O error"

I have checked and there is enough disk space available. I contacted the administrators of the cluster who said the following: "Seems it’s trying to write to an SQLite database, and that’s what’s failing. It could be something specific about the type of write they’re doing that doesn’t work on GPFS or Lustre, or it could be it’s using temporary files somewhere that has run out of space, or something else." They advised me to contact you about this issue. I'd be grateful for your advice on what could be causing this error and how to fix it.

Many thanks,

Bea

Command used and terminal output

[hzs682@rescomp1 hzs682]$ nf-core download --force -l docker.io epi2me-labs/wf-basecalling

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.10 - https://nf-co.re
    There is a new version of nf-core/tools available! (2.13.1)


? Select release / branch: v1.1.7  [release]
? Include the nf-core's default institutional configuration files into the download? Yes

If you are working on the same system where you will run Nextflow, you can amend the downloaded images to the ones in 
the$NXF_SINGULARITY_CACHEDIR folder, Nextflow will automatically find them.However if you will transfer the downloaded 
files to a different system then they should be copied to the target folder.
? Copy singularity images from $NXF_SINGULARITY_CACHEDIR to the target folder or amend new images to the cache? copy

If transferring the downloaded files to another system, it can be convenient to have everything compressed in a single 
file.
This is not recommended when downloading Singularity images, as it can take a long time and saves very little space.
? Choose compression type: none
WARNING  Deleting existing output directory: 'epi2me-labs-wf-basecalling_v1.1.7'                                       
INFO     Saving 'epi2me-labs/wf-basecalling'                                                                           
          Pipeline revision: 'v1.1.7'                                                                                  
          Use containers: 'singularity'                                                                                
          Container library: 'docker.io'                                                                               
          Using $NXF_SINGULARITY_CACHEDIR': /well/thompson/users/hzs682/singularity-cashdir'                           
          Output directory: 'epi2me-labs-wf-basecalling_v1.1.7'                                                        
          Include default institutional configuration: 'True'                                                          
INFO     Downloading centralised configs from GitHub                                                                   
INFO     Downloading workflow files from GitHub                                                                        
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/bin/nf-core:8 in <module>                │
│                                                                                                  │
│   5 from nf_core.__main__ import run_nf_core                                                     │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(run_nf_core())                                                                  │
│   9                                                                                              │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/__m │
│ ain__.py:111 in run_nf_core                                                                      │
│                                                                                                  │
│    108 │   │   │   log.debug(f"Could not check latest version: {e}")                             │
│    109 │   │   stderr.print("\n")                                                                │
│    110 │   # Launch the click cli                                                                │
│ ❱  111 │   nf_core_cli(auto_envvar_prefix="NFCORE")                                              │
│    112                                                                                           │
│    113                                                                                           │
│    114 @click.group(context_settings=dict(help_option_names=["-h", "--help"]))                   │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/click/ │
│ core.py:1130 in __call__                                                                         │
│                                                                                                  │
│   1127 │                                                                                         │
│   1128 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1129 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1130 │   │   return self.main(*args, **kwargs)                                                 │
│   1131                                                                                           │
│   1132                                                                                           │
│   1133 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/rich_click/ │
│ rich_command.py:126 in main                                                                      │
│                                                                                                  │
│   123 │   │   try:                                                                               │
│   124 │   │   │   try:                                                                           │
│   125 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                   │
│ ❱ 126 │   │   │   │   │   rv = self.invoke(ctx)                                                  │
│   127 │   │   │   │   │   if not standalone_mode:                                                │
│   128 │   │   │   │   │   │   return rv                                                          │
│   129 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                                │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/click/ │
│ core.py:1657 in invoke                                                                           │
│                                                                                                  │
│   1654 │   │   │   │   super().invoke(ctx)                                                       │
│   1655 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1656 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1657 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1658 │   │                                                                                     │
│   1659 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1660 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/click/ │
│ core.py:1404 in invoke                                                                           │
│                                                                                                  │
│   1401 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1402 │   │                                                                                     │
│   1403 │   │   if self.callback is not None:                                                     │
│ ❱ 1404 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1405 │                                                                                         │
│   1406 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1407 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/click/ │
│ core.py:760 in invoke                                                                            │
│                                                                                                  │
│    757 │   │                                                                                     │
│    758 │   │   with augment_usage_errors(__self):                                                │
│    759 │   │   │   with ctx:                                                                     │
│ ❱  760 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    761 │                                                                                         │
│    762 │   def forward(                                                                          │
│    763 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902             │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/__m │
│ ain__.py:344 in download                                                                         │
│                                                                                                  │
│    341 │   │   container_cache_index,                                                            │
│    342 │   │   parallel_downloads,                                                               │
│    343 │   )                                                                                     │
│ ❱  344 │   dl.download_workflow()                                                                │
│    345                                                                                           │
│    346                                                                                           │
│    347 # nf-core licences                                                                        │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:234 in download_workflow                                                                │
│                                                                                                  │
│    231 │   │   if self.tower:                                                                    │
│    232 │   │   │   self.download_workflow_tower()                                                │
│    233 │   │   else:                                                                             │
│ ❱  234 │   │   │   self.download_workflow_static()                                               │
│    235 │                                                                                         │
│    236 │   def download_workflow_static(self):                                                   │
│    237 │   │   """Downloads a nf-core workflow from GitHub to the local file system in a self-c  │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:248 in download_workflow_static                                                         │
│                                                                                                  │
│    245 │   │   log.info("Downloading workflow files from GitHub")                                │
│    246 │   │                                                                                     │
│    247 │   │   for item in zip(self.revision, self.wf_sha.values(), self.wf_download_url.values  │
│ ❱  248 │   │   │   revision_dirname = self.download_wf_files(revision=item[0], wf_sha=item[1],   │
│    249 │   │   │                                                                                 │
│    250 │   │   │   if self.include_configs:                                                      │
│    251 │   │   │   │   try:                                                                      │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:595 in download_wf_files                                                                │
│                                                                                                  │
│    592 │   │   log.debug(f"Downloading {download_url}")                                          │
│    593 │   │                                                                                     │
│    594 │   │   # Download GitHub zip file into memory and extract                                │
│ ❱  595 │   │   url = requests.get(download_url)                                                  │
│    596 │   │   with ZipFile(io.BytesIO(url.content)) as zipfile:                                 │
│    597 │   │   │   zipfile.extractall(self.outdir)                                               │
│    598                                                                                           │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/reques │
│ ts/api.py:73 in get                                                                              │
│                                                                                                  │
│    70 │   :rtype: requests.Response                                                              │
│    71 │   """                                                                                    │
│    72 │                                                                                          │
│ ❱  73 │   return request("get", url, params=params, **kwargs)                                    │
│    74                                                                                            │
│    75                                                                                            │
│    76 def options(url, **kwargs):                                                                │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/reques │
│ ts/api.py:59 in request                                                                          │
│                                                                                                  │
│    56 │   # avoid leaving sockets open which can trigger a ResourceWarning in some               │
│    57 │   # cases, and look like a memory leak in others.                                        │
│    58 │   with sessions.Session() as session:                                                    │
│ ❱  59 │   │   return session.request(method=method, url=url, **kwargs)                           │
│    60                                                                                            │
│    61                                                                                            │
│    62 def get(url, params=None, **kwargs):                                                       │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/requests_ca │
│ che/session.py:158 in request                                                                    │
│                                                                                                  │
│   155 │   │   """                                                                                │
│   156 │   │   headers = set_request_headers(headers, expire_after, only_if_cached, refresh, fo   │
│   157 │   │   with patch_form_boundary() if kwargs.get('files') else nullcontext():              │
│ ❱ 158 │   │   │   return super().request(method, url, *args, headers=headers, **kwargs)  # typ   │
│   159 │                                                                                          │
│   160 │   def send(                                                                              │
│   161 │   │   self,                                                                              │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/reques │
│ ts/sessions.py:587 in request                                                                    │
│                                                                                                  │
│   584 │   │   │   "allow_redirects": allow_redirects,                                            │
│   585 │   │   }                                                                                  │
│   586 │   │   send_kwargs.update(settings)                                                       │
│ ❱ 587 │   │   resp = self.send(prep, **send_kwargs)                                              │
│   588 │   │                                                                                      │
│   589 │   │   return resp                                                                        │
│   590                                                                                            │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/requests_ca │
│ che/session.py:205 in send                                                                       │
│                                                                                                  │
│   202 │   │   elif actions.resend_request:                                                       │
│   203 │   │   │   response = self._resend(request, actions, cached_response, **kwargs)  # type   │
│   204 │   │   elif actions.send_request:                                                         │
│ ❱ 205 │   │   │   response = self._send_and_cache(request, actions, cached_response, **kwargs)   │
│   206 │   │   else:                                                                              │
│   207 │   │   │   response = cached_response  # type: ignore  # Guaranteed to be non-None by t   │
│   208                                                                                            │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/requests_ca │
│ che/session.py:233 in _send_and_cache                                                            │
│                                                                                                  │
│   230 │   │   actions.update_from_response(response)                                             │
│   231 │   │                                                                                      │
│   232 │   │   if not actions.skip_write:                                                         │
│ ❱ 233 │   │   │   self.cache.save_response(response, actions.cache_key, actions.expires)         │
│   234 │   │   elif cached_response is not None and response.status_code == 304:                  │
│   235 │   │   │   cached_response = actions.update_revalidated_response(response, cached_respo   │
│   236 │   │   │   if not actions.skip_write:                                                     │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/requests_ca │
│ che/backends/base.py:91 in save_response                                                         │
│                                                                                                  │
│    88 │   │   cache_key = cache_key or self.create_key(response.request)                         │
│    89 │   │   cached_response = CachedResponse.from_response(response, expires=expires)          │
│    90 │   │   cached_response = redact_response(cached_response, self._settings.ignored_parame   │
│ ❱  91 │   │   self.responses[cache_key] = cached_response                                        │
│    92 │   │                                                                                      │
│    93 │   │   # Save redirect aliases, unless this is a revalidation (i.e., it was saved previ   │
│    94 │   │   if response.history and not cached_response.revalidated:                           │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/requests_ca │
│ che/backends/sqlite.py:309 in __setitem__                                                        │
│                                                                                                  │
│   306 │   │                                                                                      │
│   307 │   │   for i in range(self.retries):                                                      │
│   308 │   │   │   try:                                                                           │
│ ❱ 309 │   │   │   │   self._write(key, value)                                                    │
│   310 │   │   │   │   return                                                                     │
│   311 │   │   │   # Even with WAL mode, rarely a write may fail with SQLITE_BUSY; if so, retry   │
│   312 │   │   │   # self._retries times. Any other errors will be re-raised.                     │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/requests_ca │
│ che/backends/sqlite.py:331 in _write                                                             │
│                                                                                                  │
│   328 │   │   expires = getattr(value, 'expires_unix', None)                                     │
│   329 │   │   value = self.serialize(value)                                                      │
│   330 │   │   with self.connection(commit=True) as con:                                          │
│ ❱ 331 │   │   │   con.execute(                                                                   │
│   332 │   │   │   │   f'INSERT OR REPLACE INTO {self.table_name} (key,value,expires) VALUES (?   │
│   333 │   │   │   │   (key, value, expires),                                                     │
│   334 │   │   │   )                                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OperationalError: disk I/O error

System information

nextflow version 23.04.2.5871
nf-core/tools version 2.10
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.105.1.el7.x86_64
Python 3.10.8
Hardware: cluster entry node

[mashehu]

looks to me more of a problem with the caching the html request. any chance you could download the pipeline on a different location and then just copy it over?