Skip to content
This repository has been archived by the owner on Oct 16, 2023. It is now read-only.

publishDir fails to publish files to s3 when >5gb #362

Open
qscacheri opened this issue Sep 22, 2022 · 0 comments
Open

publishDir fails to publish files to s3 when >5gb #362

qscacheri opened this issue Sep 22, 2022 · 0 comments

Comments

@qscacheri
Copy link

When the result of a process is larger than 5gb, the upload to s3 fails with the error:

The specified copy source is larger than the maximum allowable size for a copy source: 5368709120 (Service: Amazon S3; Status Code: 400; Error Code: InvalidRequest

Minimal reproducible example:

nextflow.enable.dsl=2

process createFile {
    publishDir "s3://some-bucket", mode: 'copy', overwrite: true, failOnError: true
    output:
        path("*.gz")
    shell:
    '''
    truncate -s 10G test.tar.gz
    '''
}

workflow {
    createFile()
    createFile.out.view { "File: $it" }
}
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant