Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

glob pattern in filename is unexpectedly expanded #165

Open
toslunar opened this issue May 16, 2023 · 0 comments
Open

glob pattern in filename is unexpectedly expanded #165

toslunar opened this issue May 16, 2023 · 0 comments

Comments

@toslunar
Copy link

Because s3fs (fsspec) supports some glob patterns (*, **, ?, and some of []), there are file names that should not be passed to self.fs.cp or self.fs.rm without escape.

To reproduce

Example 1.

  • Add to files test_file.txt and test?file.txt.
  • A request to remove test?file.txt removes both files.

Example 2.

  • Create directory test?dir.
  • Removing (or renaming) the directory test?dir fails (by RecursionError: maximum recursion depth exceeded while calling a Python object).

To fix the issue

For now, adding glob.escape doesn't fix the issue because fsspec does not interpret the pattern [?] correctly.
https://github.com/fsspec/filesystem_spec/blob/2023.5.0/fsspec/asyn.py#L648

Instead, I configured the fs not to use glob by

class S3FileSystemNoGlob(s3fs.S3FileSystem):
    async def _expand_path(self, path, recursive=False, maxdepth=None):
        ...

and turning off the if-branch https://github.com/fsspec/filesystem_spec/blob/2023.5.0/fsspec/asyn.py#L763-L772.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant