Chunk size is not always automatically found for fs.ReadStream

[Hyzual]

Hello, first of all thank you very much for this amazing library and specification, it's very easy to use it and works well! I have found a bug while using this library in a Node.js context.

Describe the bug
The node.js installation doc says that if I pass an instance of fs.ReadStream, "no chunk will be held in memory". I took this to mean that I did not have to specify the chunkSize and uploadSize options. However, after trying to give it a stream from a fs.fileHandle, tus-js-client will complain that it needs a finite chunkSize option. I am not sure if I am misunderstanding and it was always meant to work this way or if this is a case where tus-js-client did not correctly recognize what I'm giving it.

To Reproduce
I tried hard to setup a shareable reproduction with https://codesandbox.io but I did not succeed. They run Node.js v14 and filehandle.createReadStream() was only introduced in Node.js v16. I could not find a "code sandbox" website that allows me to run Node v16. If you know of any, please share it and I'll try to setup a reproduction there.

Here is the code I used on codesandbox. It creates a HTTP server on port 8080. When called with any request, it will read a fixture.json file (its contents don't matter) and upload it to https://tusd.tusdemo.net/files/. If you comment the chunkSize or uploadSize option, tus-js-client will throw an error explaining that it needs those options.

Steps:

1. Make sure you have Node.js v16.x
2. Create an index.js file and paste the code block below into it
3. Comment chunkSize option in index.js
4. Create a package.json and install tus-js-client
5. Create a fixture.json file next to index.js and write any JSON in it, for example {"key": "value"}
6. Run the following file: node index.js
7. Call the server: curl localhost:8080
8. It should respond "An error occurred" and you should see console error

const http = require("http");
const fs = require("fs/promises");
const tus = require("tus-js-client");

const file_path = "./fixture.json"; // path to a file accessible to the node.js process. It will be uploaded to the TUS backend.
const upload_url = "https://tusd.tusdemo.net/files/"; // path to a remote TUS backend server
const CHUNK_SIZE = 67108864; // = 64 MiB. Number of bytes held in memory at a time during file upload;

function readFixtureAndUploadItToTUS() {
  return fs.open(file_path).then(async (handle) => {
    // I am forced to also run "stat" on the file to get its total size
    const file_stats = await handle.stat();

    return new Promise((resolve, reject) => {
      const uploader = new tus.Upload(handle.createReadStream({ start: 0 }), {
        endpoint: upload_url,
        // Note that tus-js-client's documentation specifies to avoid setting chunkSize and uploadSize unless forced to.
        // We are forced to set them, it does not look like the detection of ReadableStream worked in our case.
        chunkSize: CHUNK_SIZE,
        uploadSize: file_stats.size,
        onError: (error) => {
          handle.close();
          reject(error);
        },
        onSuccess: () => {
          handle.close();
          resolve();
        }
      });
      uploader.start();
    });
  });
}

http
  .createServer(function (req, res) {
    res.write("Starting the Upload !");
    readFixtureAndUploadItToTUS().then(
      () => {
        res.write("Successfully uploaded ! ");
        res.end();
      },
      (error) => {
        console.error(error);
        res.write("An error occurred");
        res.end();
      }
    );
  })
  .listen(8080);

Expected behavior
I expected not to have to set the chunkSize and uploadSize options so that tus-js-client would deduce them by itself.

Setup details

• Runtime environment: Node.js v16.17.0 (in Electron, specifically when developing a Visual Studio Code extension, if that makes any difference).
• Used tus-js-client version: 3.0.0
• Used tus server software: custom implementation in PHP, it does not seem related to the server software though.

[Acconut]

Thanks for the detailed report. tus-js-client currently relies on the path property of the ReadStream class:

tus-js-client/lib/node/fileReader.js

Line 14 in 14c3634

if (input instanceof ReadStream && input.path != null) {

It is used to get the file size via stat:

tus-js-client/lib/node/sources/FileSource.js

Line 6 in 14c3634

const { size } = await stat(path)

Since you get it from an file descriptor, this property is not present and tus-js-client falls back to handling it as any arbitrary readable stream, which requires buffer memory and the chunkSize option.

I think in theory we could use fstat to get the size via the file descriptor, if it is present but I haven't tested that yet. Would you be interested in submitting a PR?

[Hyzual]

Thanks for the very quick answer ! Indeed maybe fstat could be used to auto-detect the file size. I'm sorry, I don't think I will have much time to commit to getting into the codebase and submitting a PR. I might try it on my free time though.

[Acconut]

I'm sorry, I don't think I will have much time to commit to getting into the codebase and submitting a PR. I might try it on my free time though.

I do understand that! I am also not sure when I have time to address this.