Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs_scandir_next() sometimes returns name but no type #660

Open
troiganto opened this issue Jul 25, 2023 · 6 comments
Open

fs_scandir_next() sometimes returns name but no type #660

troiganto opened this issue Jul 25, 2023 · 6 comments
Labels

Comments

@troiganto
Copy link

Hi!

After debugging for a long time why the LuaSnip plugin for Neovim doesn't load all my snippets, I think I've traced the issue to this (otherwise excellent!) library.

The documentation for luv.fs_scandir_next() states that the function either returns two strings (file name and directory entry type) or nil or it fails completely.

However, for whatever reason, on the specific machine that I use, I receive the file name as usual but nil for the entry type. (MWE at the bottom.) Because downstream Lua scripts expect either two strings or two nils, this breaks a lot of their assumptions and code starts to behave strangely.

After some digging, I think I located the issue in fs.c:126: whenever ent->type is an unknown enum value, the type is set to the string "unknown" and two values are returned as usual. However, if ent->type is the known enum value UV_DIRENT_UNKNOWN, then the type is not set at all and only one value is returned.

I'm not sure what the correct behavior is here, but I think either the code should be changed (by merging the UV_DIRENT_UNKNOWN with the default) or the docs (to inform users that sometimes string, nil is returned).

Of course, if you can think of a simple reason why the directory entry type suddenly fails to be recognized, I'd be extremely grateful for any pointers. 🙂

No matter how you decide, thanks for your time and hard work!

Minimal example (though I'm not sure how useful it is on most machines):

$ git clone https://github.com/luvit/luv.git --recursive
...
$ cd luv
$  git describe
1.40.0-0-148-gd15a473
$ make
...
$ cd build
$ ./luajit ~/mwe.lua

where mwe.lua:

function Main()
  local luv = require "luv"
  local root = "<HOME>/.local/share/nvim/lazy/vim-snippets/snippets"
  local fs = luv.fs_scandir(root)
  local name, type = "", ""
  while name do
    name, type = luv.fs_scandir_next(fs)
    print(type, name)
  end
end

Main()

and the output is:

file	_.snippets
file	actionscript.snippets
file	ada.snippets
file	all.snippets
file	alpaca.snippets
file	apache.snippets
file	arduino.snippets
file	asm.snippets
file	autoit.snippets
file	awk.snippets
file	bash.snippets
file	c.snippets
file	chef.snippets
file	clojure.snippets
file	cmake.snippets
file	codeigniter.snippets
directory	coffee
file	cpp.snippets
nil	crystal.snippets
nil	cs.snippets
nil	css.snippets
nil	cuda.snippets
nil	d.snippets
nil	dart-flutter.snippets
nil	dart.snippets
nil	diff.snippets
nil	django.snippets
nil	dosini.snippets
nil	eelixir.snippets
nil	elixir.snippets
nil	elm.snippets
nil	erlang.snippets
nil	eruby.snippets
nil	falcon.snippets
nil	fortran.snippets
nil	freemarker.snippets
nil	fsharp.snippets
nil	gdscript.snippets
nil	gitcommit.snippets
nil	gleam.snippets
nil	go.snippets
nil	haml.snippets
nil	handlebars.snippets
nil	haskell.snippets
nil	heex.snippets
nil	helm.snippets
nil	html.snippets
nil	htmldjango.snippets
nil	htmltornado.snippets
nil	idris.snippets
nil	jade.snippets
nil	java.snippets
nil	javascript
nil	javascript-bemjson.snippets
nil	javascript-d3.snippets
nil	javascript-jasmine.snippets
nil	javascript-mocha.snippets
nil	javascript-openui5.snippets
nil	jenkins.snippets
nil	jinja.snippets
nil	jsp.snippets
nil	julia.snippets
nil	kotlin.snippets
nil	laravel.snippets
nil	ledger.snippets
nil	lfe.snippets
nil	liquid.snippets
nil	lpc.snippets
nil	ls.snippets
nil	lua.snippets
nil	make.snippets
nil	mako.snippets
nil	markdown.snippets
nil	matlab.snippets
nil	mustache.snippets
nil	objc.snippets
nil	ocaml.snippets
nil	octave.snippets
nil	openfoam.snippets
nil	org.snippets
nil	pandoc.snippets
nil	perl.snippets
nil	perl6.snippets
nil	phoenix.snippets
nil	php.snippets
nil	plsql.snippets
nil	po.snippets
nil	processing.snippets
nil	progress.snippets
nil	ps1.snippets
nil	puppet.snippets
nil	purescript.snippets
nil	python.snippets
nil	r.snippets
nil	racket.snippets
nil	rails.snippets
nil	reason.snippets
nil	rmd.snippets
nil	rst.snippets
nil	ruby.snippets
nil	rust.snippets
nil	sass.snippets
nil	scala.snippets
nil	scheme.snippets
nil	scss.snippets
nil	sh.snippets
nil	simplemvcf.snippets
nil	slim.snippets
nil	smarty.snippets
nil	snippets.snippets
nil	sql.snippets
nil	stylus.snippets
nil	supercollider.snippets
nil	svelte.snippets
nil	systemverilog.snippets
nil	tcl.snippets
nil	tex.snippets
nil	textile.snippets
nil	twig.snippets
nil	typescript.snippets
nil	typescriptreact.snippets
nil	verilog.snippets
nil	vhdl.snippets
nil	vim.snippets
nil	vue.snippets
nil	xml.snippets
nil	xslt.snippets
nil	yii-chtml.snippets
nil	yii.snippets
nil	zsh.snippets
nil	nil
@SinisterRectus
Copy link
Member

Odd that it works then stops working.

According to https://www.gnu.org/software/libc/manual/html_node/Directory-Entries.html:

The type is unknown. Only some filesystems have full support to return the type of the file, others might always return this value.

On what OS and filesystem do you see this happening?

@troiganto
Copy link
Author

It's odd indeed, it took me a long time to piece together what was happening here.

I access the affected machine remotely, it's managed by our IT department. I'm in contact with them in parallel to figure out if there's something they can do.

$ cat /etc/redhat-release
Rocky Linux release 9.2 (Blue Onyx)
$ mount | grep "/home "
<...> on /home type nfs4 (rw,nosuid,nodev,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=<ip addr>,local_lock=none,addr=<ip addr>)

I wonder if the network filesystem and its inevitable lag cause any shenanigans somewhere inside libuv …

@squeek502
Copy link
Member

Surprised that our docs don't mention this; that should definitely be fixed regardless of whether we return nil or "unknown" as the type.

I assume the same applies to fs_readdir but the Libuv docs don't mention it there. Will need to try to test that as well.

@Bilal2453
Copy link
Contributor

We can make this return a "unknown" string as well, this is probably the more correct behavior here. I remember reading the code relating this, should be an easy fix, assuming it wasn't intended behavior this shouldn't be a breaking change as well.

@squeek502
Copy link
Member

Personally, I think I prefer keeping it as nil, or at least keeping some way of distinguishing between 'we got a type but it's unknown--it is not one of the known types though' and 'we didn't get a type at all--it could be anything, you probably need to do a stat call to find out for sure'.

@troiganto
Copy link
Author

I can confirm that fs_readdir() shows the same behavior:

function Main()
  local luv = require "luv"
  local root = "<home>/.local/share/nvim/lazy/vim-snippets/snippets"
  local fs = luv.fs_opendir(root, nil, 200)
  local data = luv.fs_readdir(fs)
  if data then
    for _, dirent in ipairs(data) do
      print(dirent.type, dirent.name)
    end
  end
  if luv.fs_readdir(fs) then
    print("...")
  end
  luv.fs_closedir(fs)
end

Main()

This prints the same (lengthy) output as the original example.

Explicitly using stat on the entries that couldn't be read seems to kick the filesystem into doing its job. The following script correctly marks all files as "file" instead of nil:

function Main()
  local luv = require "luv"
  local root = "<home>/.local/share/nvim/lazy/vim-snippets/snippets"
  local fs = luv.fs_scandir(root)
  local name, type = "", ""
  while name do
    name, type = luv.fs_scandir_next(fs)
    if name and not type then
      local stat = luv.fs_stat(root .. "/" .. name)
      type = stat.type
    end
    print(type, name)
  end
end

Main()

troiganto pushed a commit to troiganto/LuaSnip that referenced this issue Jul 26, 2023
While libuv/vim.loop docs say that `fs_scandir_next()` either returns
`nil` or `string, string` or fails, there is an edge case on networked
filesystems. In those cases, `uv__fs_scandir()` returns the type
`UV_DIRENT_UNKNOWN` and `luv.fs_scandir_next()` [converts][1] this into
returning a single string.

This means in those cases, `name` is a string and `type` is `nil`. See
[this bug report][2].

The situation can be remedied by explicitly calling `fs_stat()` on those
files. This always fetches the correct type, as far as I can see.

[1]: https://github.com/luvit/luv/blob/master/src/fs.c#L116
[2]: luvit/luv#660
L3MON4D3 pushed a commit to L3MON4D3/LuaSnip that referenced this issue Jul 26, 2023
While libuv/vim.loop docs say that `fs_scandir_next()` either returns
`nil` or `string, string` or fails, there is an edge case on networked
filesystems. In those cases, `uv__fs_scandir()` returns the type
`UV_DIRENT_UNKNOWN` and `luv.fs_scandir_next()` [converts][1] this into
returning a single string.

This means in those cases, `name` is a string and `type` is `nil`. See
[this bug report][2].

The situation can be remedied by explicitly calling `fs_stat()` on those
files. This always fetches the correct type, as far as I can see.

[1]: https://github.com/luvit/luv/blob/master/src/fs.c#L116
[2]: luvit/luv#660
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants