mkdir is failing with the following error #272
Comments
|
has anyone seen this problem? thanks. |
|
Hi This is the command I used hdfs dfs -mkdir cos://hdfsVault.applogs/1/ 2021-03-26 17:00:29,873 INFO fs.ObjectStoreVisitor: Stocator registered as cos for cos://hdfsVault.applogs/1 |
|
Hi @nkurn
|
|
Hi @mrmadira OSNAME="Ubuntu" hadoopHadoop 3.2.2 stocatorstocator-1.1.4 did you see the core-site.xml I have added? thanks. |
|
hdfs dfs -ls |
|
WE don't have a repro for this. Please close until we get more details |
|
Hi, why is stocator checking for file status ": file status 1 returned 404" when creating a directory? thanks a lot, |
|
Also, scala> val statuses = fs.mkdirs(new Path(s"cos://TestVault1.mycos/d6")) it is failing only when I use the shell command, do you think I am missing some hadoop/hdfs configuration? |
|
hdfs dfs -mkdir cos://TestVault1.mycos/d7 |
|
Hi Naga -can you share your classpath that takes effect for HDFS? |
|
HADOOP_CLASSPATH=/src/hadoop-release/lib/:/src/hadoop-release/etc/hadoop/conf/:/src/hadoop-release/etc/hadoop/:/src/hadoop-release/share/hadoop/tools/lib/ |
|
I am the root user when I run the shell FS commands, does it matter which user runs hdfs mkdir commands? |
|
on the spark-shell using fs.mkdirs call the accesser is getting the put request and is succeeding {"server_name":"xxxx","remote_address":"xxxx","remote_user":"xxxx","timestamp_start":"1622645396705","timestamp_finish":"1622645397190","time_start":"02/Jun/2021:14:49:56 +0000","time_finish":"02/Jun/2021:14:49:57 +0000","request_method":"PUT","request_uri":"/TestVault1/tdir1/","protocol":"HTTP/1.1","status":200,"response_length":"0","request_length":"0","user_agent":"stocator 1.0.38-SNAPSHOT, aws-sdk-java/1.11.589 Linux/3.10.0-1160.21.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242 scala/2.11.12 vendor/Amazon.com_Inc. com.amazonaws.services.s3.transfer.TransferManager/1.11.589","request_latency":"485","request_id":"51dabdce-9374-472d-ac6f-86e70a91c1a8","request_type":"REST.PUT.OBJECT","interface_type":"s3","stat":{"client_wait":0.05,"storage_wait":0.491,"digest":0.0,"commit":460.31,"turn_around_time":0.0,"total_transfer":460.862,"pre_transfer":22.681,"post_transfer":0.321},"object_length":"0","version_name":"da7c586a-40c5-4f4d-ad95-a4b8db4946f7","version_transient":true,"delete_marker":false,"last_modified":"2021-06-02T14:49:56.729Z","last_changed":"2021-06-02T14:49:56.729Z","e_tag":"d41d8cd98f00b204e9800998ecf8427e","object_name":"tdir1/","vault_name":"TestVault1","is_secure":false,"principals":{"identity":"1d86d781-35fa-44a2-b327-322e3cb8723c@00000000-0000-0000-0000-000000000000","aws":"xxx"},"type":"http","format":1} |
|
when it fails on the FS shell there are 3 requests "REST.HEAD.OBJECT, REST.HEAD.OBJECT and REST.GET.VAULT {"server_name":"xxx","remote_address":"xxx","remote_user":"xxx","timestamp_start":"1622645524767","timestamp_finish":"1622645524831","time_start":"02/Jun/2021:14:52:04 +0000","time_finish":"02/Jun/2021:14:52:04 +0000","request_method":"HEAD","request_uri":"/TestVault1/tdir2","protocol":"HTTP/1.1","status":404,"response_length":"277","user_agent":"stocator 1.0.38-SNAPSHOT, aws-sdk-java/1.11.199 Linux/3.10.0-1160.21.1.el7.x86_64 OpenJDK_64-Bit_Server_VM/25.242-b08 java/1.8.0_242","request_latency":"64","request_id":"adf2cd5c-f73b-4841-be27-6ad5a24b0a40","request_type":"REST.HEAD.OBJECT","interface_type":"s3","object_name":"tdir2","error_code":"NoSuchKey","vault_name":"TestVault1","is_secure":false,"principals":{"identity":"1d86d781-35fa-44a2-b327-322e3cb8723c@00000000-0000-0000-0000-000000000000","aws":"xxx"},"type":"http","format":1} |
|
user_agent seems to be a little different each time, can you please check the requests above and see if it makes sense |
|
Seems you are on a very old version of stocator. Can you please try with latest version 1.1.3? |
|
I think you will face a different behaviour on latest though; I can see it throws a FNF exception on latest. |
|
I tested the latest as well, it also fails. thanks a lot for the workaround !! it is working. Please let me know once you have something for me to test. Thanks again for your help !! |
|
nested directories creation doesn't seem to work hdfs dfs -mkdir /dir2/dir3 is there a way to make it work? thanks, |
|
looks like I need to use 2 step process hdfs dfs -mkdir /dir2 |
2021-03-03 23:27:05,940 INFO fs.ObjectStoreVisitor: Stocator registered as cos for cos://hdfsVault.mycos/dir1
2021-03-03 23:27:05,952 INFO cos.COSAPIClient: Init : cos://hdfsVault.mycos/dir1
2021-03-03 23:27:07,195 WARN cos.COSAPIClient: file status dir1 returned 404
-mkdir: Fatal internal error
java.lang.NullPointerException
at com.ibm.stocator.fs.common.ObjectStoreGlobber.glob(ObjectStoreGlobber.java:180)
at com.ibm.stocator.fs.ObjectStoreFileSystem.globStatus(ObjectStoreFileSystem.java:544)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:353)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
my core-site.xml
fs.stocator.scheme.list cos fs.cos.impl com.ibm.stocator.fs.ObjectStoreFileSystem fs.stocator.cos.impl com.ibm.stocator.fs.cos.COSAPIClient fs.stocator.cos.scheme cos fs.cos.mycos.v2.signer.type false fs.cos.mycos.endpoint xx fs.cos.mycos.access.key xx fs.cos.mycos.secret.key xxThe text was updated successfully, but these errors were encountered: