Upper Limit to input file size?

[collinwhite]

Hello all,

I am trying to use csvlink across two input files as normal. However, one of the csv files I'm using is several GB in size (how the file is given to me, I have no control over this).

If I attempt to run csvlink with this file, I get the following

error: Could not find the file <file>

Upon investigating the issue, the file is of course there. I took time to break it out into smaller files as I assumed perhaps this issue was being caused by the file size. This appears to be the case. If I attempt to run csvlink with a smaller (still large, 750MB) file, it is running. Hasn't finished yet, but it's running ;)

I'm reporting this issue to let someone at the team know about it. If there is a designed upper limit to the file size that csvdedupe/csvlink can use, let me know here and I can open a PR with changes to the README.

Thanks!