Error marker_xp->nodesetval is empty

[sherpaurgen]

Version of s3fs being used (s3fs --version)

s3fs version 1.91

Version of fuse being used (pkg-config --modversion fuse, rpm -qi fuse or dpkg -s fuse)

Fuse v2.9.9-3

Kernel information (uname -r)

5.4.0-125-generic

GNU/Linux Distribution, if applicable (cat /etc/os-release)

(ubuntu 20.04.4)

How to run s3fs, if applicable , cat /etc/fstab . Modified parts below

mybucket:/bucketname   /path/to/destination   fuse.s3fs _netdev,nonempty,allow_other,nosuid,nodev,uid=1234,gid=1234,umask=7022,passwd_file=/some/path,url=https://s3.us-west-1.domain.com,logfile=/path/to/debug.log,dbglevel=info   0 0

s3fs syslog messages

2023-12-28T16:07:11.272Z [ERR] fdcache_entity.cpp:Read(1750): could not reserve disk space for pre-fetch download
2023-12-28T16:07:11.272Z [WAN] s3fs.cpp:s3fs_read(2345): failed to read file(/bucket/smallfiles/loadsOf/12/06/1701820800/_yv7.cfs.bgz.gz). result=-28
2023-12-28T16:07:11.272Z [WAN] fdcache_entity.cpp:Read(1747): could not reserve disk space for pre-fetch download
2023-12-28T16:07:11.273Z [ERR] fdcache_entity.cpp:Read(1750): could not reserve disk space for pre-fetch download

2024-01-01T01:18:40.920Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty.
2024-01-01T01:18:40.922Z [INF]       curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01]
2024-01-01T01:18:40.922Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:40.922Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:40.922Z [INF]       curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701415800_3.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] []
2024-01-01T01:18:40.922Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com
2024-01-01T01:18:41.013Z [INF]       curl.cpp:RequestPerform(2327): HTTP response code 200
2024-01-01T01:18:41.023Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty.
2024-01-01T01:18:41.024Z [INF]       curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01]
2024-01-01T01:18:41.024Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:41.025Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:41.025Z [INF]       curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701443100_3.gz&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] []
2024-01-01T01:18:41.025Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com
2024-01-01T01:18:41.108Z [INF]       curl.cpp:RequestPerform(2327): HTTP response code 200
2024-01-01T01:18:41.118Z [ERR] s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty.
2024-01-01T01:18:41.119Z [INF]       curl.cpp:ListBucketRequest(3525): [tpath=/mydata/myserverlog/2023/12/01]
2024-01-01T01:18:41.119Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:41.119Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com?delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/
2024-01-01T01:18:41.119Z [INF]       curl.cpp:insertV4Headers(2699): computing signature [GET] [/] [delimiter=/&marker=myserverlog/mydata/myserverlog/2023/12/01/1701470400_2.sha1&max-keys=1000&prefix=myserverlog/mydata/myserverlog/2023/12/01/] []
2024-01-01T01:18:41.119Z [INF]       curl_util.cpp:url_to_host(334): url is https://s3.us-east-2.wasabi.com
2024-01-01T01:18:41.142Z [INF]       curl.cpp:RequestPerform(2327): HTTP response code 200
2024-01-01T01:18:41.144Z [INF]   s3fs.cpp:readdir_multi_head(2586): [path=/mydata/myserverlog/2023/12/01/][list=0]
2024-01-01T01:18:41.145Z [INF]       cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800.prop]
2024-01-01T01:18:41.146Z [INF]       curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800.prop][bpath=1701388800.prop][save=/mydata/myserverlog/2023/12/01/1701388800.prop][sseckeypos=18446744073709551615]
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800.prop
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800.prop
2024-01-01T01:18:41.146Z [INF]       cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_0.gz]
2024-01-01T01:18:41.146Z [INF]       curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_0.gz][bpath=1701388800_0.gz][save=/mydata/myserverlog/2023/12/01/1701388800_0.gz][sseckeypos=18446744073709551615]
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.gz
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.gz
2024-01-01T01:18:41.146Z [INF]       cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_0.sha1]
2024-01-01T01:18:41.146Z [INF]       curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_0.sha1][bpath=1701388800_0.sha1][save=/mydata/myserverlog/2023/12/01/1701388800_0.sha1][sseckeypos=18446744073709551615]
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.sha1
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_0.sha1
2024-01-01T01:18:41.146Z [INF]       cache.cpp:DelStat(597): delete stat cache entry[path=/mydata/myserverlog/2023/12/01/1701388800_1.gz]
2024-01-01T01:18:41.146Z [INF]       curl.cpp:PreHeadRequest(3063): [tpath=/mydata/myserverlog/2023/12/01/1701388800_1.gz][bpath=1701388800_1.gz][save=/mydata/myserverlog/2023/12/01/1701388800_1.gz][sseckeypos=18446744073709551615]
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(257): URL is https://s3.us-east-2.wasabi.com/prods-mydata/myserverlog/mydata/myserverlog/2023/12/01/1701388800_1.gz
2024-01-01T01:18:41.146Z [INF]       curl_util.cpp:prepare_url(290): URL changed is https://prods-mydata.s3.us-east-2.wasabi.com/myserverlog/mydata/myserverlog/2023/12/01/1701388800_1.gz

Details about issue

For some reason the s3fs gets un-mounted and nothing is logged in syslog/dmesg or in journalctl, Therefore i have enabled logging in fstab mount option.
What could be the reason for the ERROR message s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty. ?

There are entries in logs that says "could not reserve diskspace for prefetch download" but the use_cache is default/disabled.
https://manpages.ubuntu.com/manpages/focal/man1/s3fs.1.html

[tanguofu]

same question

[ggtakec]

Your problem seems unrelated to the error message s3fs_xml.cpp:get_base_exp(110): marker_xp->nodesetval is empty..
Actually, this message shouldn't be at ERR level, so I think I'll create a new PR.
I think this message was confusing and confused you.

The problem with this issue seems to be the could not reserve disk space for pre-fetch download error message that you also noticed.
To read files, s3fs stores downloaded objects as temporary files on the local disk.
I get an error saying there isn't enough space for this.
Note that the use_cache option keeps and caches this temporary file for a certain period of time. This does not mean that this temporary file is not created. So you need free space for download.