Skip to content
This repository has been archived by the owner on Apr 19, 2022. It is now read-only.

python snakebite module dosen't handle special character #251

Open
hackaholic opened this issue Jun 7, 2018 · 1 comment
Open

python snakebite module dosen't handle special character #251

hackaholic opened this issue Jun 7, 2018 · 1 comment

Comments

@hackaholic
Copy link

On below directories when snakebite python module tries to do ls recursively it, returns no such file or directory.


mybox]$ hdfs dfs -ls /data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/ | grep Tenant
drwxr-xr-x   - tdcdv1r tdcdv1c          0 2018-05-01 18:28 /data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/rtng_ky=Access.NBNOrder.Amend.Info.{Tenant}.Rejected.v2.event
drwxr-xr-x   - tdcdv1r tdcdv1c          0 2018-05-01 15:35 /data/tdc/dv1/corp/base/dpp/raw/load_date=2018-05-01/rtng_ky=Access.NBNOrder.Amend.Info.{Tenant}.v2.event

I know its due to {} special character but snakebite is not able to handle it, does any one have any solution for this problem?.

Below is the python code:

from snakebite.client import HAClient
from snakebite.namenode import Namenode
name_node_list = [Namenode('namenode-01.localhost.com', 8020),
Namenode('namenode-02.localhost.com', 8020)]
hdfs_namenode_principal = 'nn/namenode-01.localhost.com@localhost.com'
client = HAClient(name_node_list, use_trash=False, use_sasl=True, hdfs_namenode_principal=hdfs_namenode_principal)
for x in client.ls([path], recurse=True):
    print x

I have also posted the same on stackoverflow: https://stackoverflow.com/questions/50733649/python-snakebite-module-ls-return-error-no-such-file-or-directory

@kockockockoc
Copy link

Yeh, same issue:

# hdfs dfs -ls /tmp | grep "{"
drwxr-xr-x   - test            hadoop           0 2018-08-21 18:54 /tmp/test{111,222}

I temporarily commented out glob.expand_paths():

--- snakebite/client.py.orig   2018-08-21 19:03:24.278308337 +0300
+++ snakebite/client.py        2018-08-21 18:56:22.275410644 +0300
@@ -1268,7 +1268,7 @@
             paths = [posixpath.join("/user", get_current_username())]

         # Expand paths if necessary (/foo/{bar,baz} --> ['/foo/bar', '/foo/baz'])
-        paths = glob.expand_paths(paths)
+        # paths = glob.expand_paths(paths)

         for path in paths:
             if not path.startswith("/"):

But I'm not sure that I'm right. There may be some side effects

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants