fix: allow files in directories to be downloaded onto local machine

[hochoy]

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary) (n/a)

Fixes #2200 🦕

[hochoy]

@ddelgrosso1 , I tried running samples-test and system-test but based on the logs, it appears those require an active project in order to do an e2e/live test against real GCP infrastructure.

Is that accurate?
How expensive is that if I run it on my own project? Should I be triggering it here in the repo instead (and leverage your repo's cloudbuild/github actions infra)?

[hochoy]

Also, I would like to add a test to check that the files are truly downloaded into the local machine or memory.
Would you recommend I give that a shot? Or is that overkill?

[ddelgrosso1]

@ddelgrosso1 , I tried running samples-test and system-test but based on the logs, it appears those require an active project in order to do an e2e/live test against real GCP infrastructure.

Is that accurate? How expensive is that if I run it on my own project? Should I be triggering it here in the repo instead (and leverage your repo's cloudbuild/github actions infra)?

That is accurate. I wouldn't worry about running these on your own, they get run in the CI pipeline each time a commit is pushed. However, unit tests should work without issue locally.

Also, I would like to add a test to check that the files are truly downloaded into the local machine or memory. Would you recommend I give that a shot? Or is that overkill?

If you feel up to it a unit test can probably be created to test this. I can look to see if we have any similar tests elsewhere that might serve as a guide.

One thing I will do is to cleanup the the JS Docs to make it abundantly clear that not supplying a prefix will result in the files downloaded to memory.

[hochoy]

Any assert(false) will throw an Error, meaning done() never gets called and the test times out. Adding the try catch guarantees done() gets called. Similar to what is described in this Stack overflow discussion: https://stackoverflow.com/questions/66461468/mocha-test-false-assert-timeouts

I'm a bit surprised why this doesn't cause issues in the fs.readFile callback assertions in the other tests.

[ddelgrosso1]

Interesting find, let me dig into this a little bit more.

[ddelgrosso1]

LGTM

[ddelgrosso1]

@danielbankhead would you mind just giving this a second set of eyes?

[danielbankhead]

Looks great! I think we can make it better for File#download customers by moving the logic to TransferManager#downloadManyFiles

[danielbankhead]

I think this could be problematic as / could be a valid ending character for an object name: https://cloud.google.com/storage/docs/objects

Customers, namely outside of the Transfer Manager flow, could face issues.

[hochoy]

Hey @danielbankhead , thanks for the review. The issue with object names ending with / is that while they may be valid on GCS, it can't actually be written on a Linux-based filesystem. I don't have a windows to try it on. Perhaps you know of an edge case where names ending with / can actually be written.

[hochoy]

The scenario you described is valid, if destination is unset/falsy, and a Buffer gets created/returned for the GCS object ending with /. That behaviour remains unchanged in this PR

[danielbankhead]

For this case, I think allowing the file system to error would be clearer and easier to understand than returning a Buffer.

[ddelgrosso1]

@vishwarajanand I think we want to address this per @danielbankhead feedback.

[danielbankhead]

I think this logic should live out of File#download for a few reasons:

• The mkdirSync would make this method slower for non-TM customers:
  • where the directory already exist (unnecessary I/O)
  • with slower filesystems (blocking I/O)
• The logic for determining if a directory should be created can be handled for multiple files at once (rather than each file in the same directory doing the same work).

[hochoy]

Coincidentally, that was the original design I had (creating the directory structure on TransferManager). However, I made the change on File#download because the root issue is that File#download itself cannot write nested paths. And solving it at the lower level would solve the problem for both File and TransferManager.

That being said, it's not hard to optimize at TransferManager, I can create another branch. What do we do then about the limitation of File#download above? Do we:

1. warn users not to provide nested destination paths?
2. or warn users that they need to ensure that the folder hierarchy exists before providing nested destination paths?

[danielbankhead]

I think a simple solution would be to add a flag to gate this functionality and Transfer Manager use this flag.

[ddelgrosso1]

@vishwarajanand same comment here, I think we want to address @danielbankhead idea about flagging / gating this.

[hochoy]

Hey @danielbankhead @ddelgrosso1 , just wanted to provide a heads up that I'll be revisiting this in about 1-2 months due to project priorities. Thank you for all the feedback and recommendations, they are all valid points for a user like myself.

I do need something like TransferManager for a project at work, so I'll very likely be back (no promises of course). But in its current design, it would be more practical for me to build my own version of the TransferManager to reflect the interface I'm looking for. I will try to wrap the existing File.download method during my experimentation, similar to the current design, but might go a different way if needed.

Couple things I would design for, after both your feedback:

• downloads should always be concurrent
• empty objects (like "folders") on gcp are not writeable need special handling
• directory creation should be optimized at the TransferManager level
• unix vs windows-based file handling
• read and write failures should not fail silently. In the event of a partial directory/folder download, we should make it easy for the user to identify which files succeeded vs failed.
• if similar files exist in the write destination, we could provide a default of either over-writing or error-throwing or user-prompt

I envision users wanting to use TransferManager to download entire buckets or "sub-directories". So, those are some of the considerations I could think of.

Something else that the Storage service should consider, is whether this should be done at the API-level instead. Allowing users to use more advanced queries. I know adding any logic around bucket objects is likely terrible for performance, but it would make this cross-language compatible. Trying to implement TransferManager across all supported languages would mean a lot of SDK work.