Nimble install nimlsp cannot open file /usr/nimsuggest/nimsuggest.nim

[wbehrens-on-gh]

This issue doesn't come up when installing other nimble packages (nake, godot, nimx, etc).

When I run nimble install nimlsp it returns the following error:

I have nim and nimble installed using the package inside the pacman repos.
versions:
nim: 1.4.2
nimble: 0.12.0

[SolitudeSF]

duplicate #67

[PMunch]

As SolitudeSF mentioned this is a duplicate. But maybe NimLSP should detect this case and give a better warning so that it's easier for users to figure out what's wrong

[SolitudeSF]

nimsuggest should be a nimble package

[PMunch]

If you install the compiler package is has nimsuggest as part of its sources. The problem is just that NimLSP needs to be compiled with sources matching the Nim version you want to use. I have polled @dom96 to add a hooks feature to choosenim so NimLSP can be built with the currently chosen choosenim version, but it's currently tagged as "Low priority" and has been for quite a while (even though it could just be as easy as having a single script file that is run whenever choosenim completes.

The alternative is for NimLSP to be choosenim aware. So it would check when run whether the choosenim command was available and in that case either switch version or recompile, otherwise it would do it's normal thing. It means that NimLSP needs to know how to re-build itself though, and might not be optimal.

Araq also suggested putting NimLSP into the core Nim distribution which would solve all of this. But that would mean the Nim tools would have external dependencies, which is a bit odd.

[dimik]

How to build it on mac?

nimble install -d:explicitSourcePath:./ nimlsp
Downloading https://github.com/PMunch/nimlsp using git
  Verifying dependencies for nimlsp@0.3.2
      Info: Dependency on astpatternmatching@any version already satisfied
  Verifying dependencies for ast_pattern_matching@1.0.0
      Info: Dependency on jsonschema@>= 0.2.1 already satisfied
  Verifying dependencies for jsonschema@0.2.1
      Info: Dependency on ast_pattern_matching@any version already satisfied
  Verifying dependencies for ast_pattern_matching@1.0.0
 Installing nimlsp@0.3.2
   Building nimlsp/nimlsp using c backend
/private/var/folders/kc/6_74dq6s5_v83v8cw5lr6hmh0000gn/T/nimble_31506/githubcom_PMunchnimlsp/src/nimlsppkg/suggestlib.nim(11, 8) template/generic instantiation of `mImport` from here
/usr/local/Cellar/nim/1.4.6/nim/nimsuggest/sexp.nim(15, 19) Error: cannot open file: std/private/decode_helpers

[PMunch]

I'm not sure if you can pass command arguments like that to Nimble, or it might just not like the relative path. I'd recommend installing Nim via choosenim, but if that's not an option then you might have to install NimLSP manually to build with explicitSourcePath

[dimik]

@PMunch thank you for instant reply. Nim is installed with no problem using choosenim or homebrew, but nimble doesn't know anything about nimlsp, so i tried the manual build. It requires nimsuggest sources. So i have to check out the Nim repo to have nimsuggest/nimsuggest.nim

Actually the question is how to install it manually on mac? It depends on nimsuggest which tries to include "std/private/decode_helpers" and fails. This one is with "choosenim" but result is the same

 ~/nimlsp> nimble build -d:explicitSourcePath:/Users/dp/Nim
  Verifying dependencies for nimlsp@0.3.2
      Info: Dependency on astpatternmatching@any version already satisfied
  Verifying dependencies for ast_pattern_matching@1.0.0
      Info: Dependency on jsonschema@>= 0.2.1 already satisfied
  Verifying dependencies for jsonschema@0.2.1
      Info: Dependency on ast_pattern_matching@any version already satisfied
  Verifying dependencies for ast_pattern_matching@1.0.0
   Building nimlsp/nimlsp using c backend
/Users/dp/nimlsp/src/nimlsppkg/suggestlib.nim(11, 8) template/generic instantiation of `mImport` from here
/Users/dp/Nim/nimsuggest/sexp.nim(15, 19) Error: cannot open file: std/private/decode_helpers
       Tip: 7 messages have been suppressed, use --verbose to show them.
     Error: Build failed for package: nimlsp
        ... Execution failed with exit code 256
        ... Command: /Users/dp/.nimble/bin/nim c --colors:on --noNimblePath -d:explicitSourcePath:/Users/dp/Nim -d:NimblePkgVersion=0.3.2 --path:/Users/dp/.nimble/pkgs/ast_pattern_matching-1.0.0 --path:/Users/dp/.nimble/pkgs/jsonschema-0.2.1 --path:/Users/dp/.nimble/pkgs/ast_pattern_matching-1.0.0 --hints:off -o:/Users/dp/nimlsp/nimlsp /Users/dp/nimlsp/src/nimlsp.nim

[dimik]

I managed to do it finally. Not sure if it's a proper way or too hacky. I moved nimlsp sources folder to Nim sources folder, and made couple changes to koch.nim file. This is my diff in case somebody stuck with the same issue:

diff --git a/koch.nim b/koch.nim
index 15355ec36..6c69e8619 100644
--- a/koch.nim
+++ b/koch.nim
@@ -169,6 +169,10 @@ proc bundleNimpretty(args: string) =
   nimCompileFold("Compile nimpretty", "nimpretty/nimpretty.nim",
                  options = "-d:release " & args)

+proc bundleNimlsp(args: string) =
+  nimCompileFold("Compile nimlsp", "nimlsp/src/nimlsp.nim",
+                 options = "-d:release " & args)
+
 proc bundleWinTools(args: string) =
   nimCompile("tools/finish.nim", outputDir = "", options = args)

@@ -185,6 +189,7 @@ proc zip(latest: bool; args: string) =
   bundleNimbleExe(latest, args)
   bundleNimsuggest(args)
   bundleNimpretty(args)
+  bundleNimlsp(args)
   bundleWinTools(args)
   nimexec("cc -r $2 --var:version=$1 --var:mingw=none --main:compiler/nim.nim scripts compiler/installer.ini" %
        [VersionAsString, compileNimInst])
@@ -215,6 +220,7 @@ proc buildTools(args: string = "") =
                  options = "-d:release " & args)
   when defined(windows): buildVccTool(args)
   bundleNimpretty(args)
+  bundleNimlsp(args)
   nimCompileFold("Compile testament", "testament/testament.nim",
                  options = "-d:release " & args)

(END)

There was a warning during compile time, @PMunch not sure if it's important:

/Users/dp/Nim/nimlsp/src/nimlsppkg/baseprotocol.nim(26, 29) Warning: Deprecated since 1.5; TaintedString is deprecated [Deprecated]

After that just run build_all.sh and it compiles to Nim/bin/nimlsp
Not sure why you can't just move it to the main repo the same way and just simplify the things

[pjz]

I can kind of understand why nimlsp needs access to the nim sources, but why not just have it carry a copy of the correct source for every major release of nim (handles common cases), along with a way to accept a path to an external source repo (handles the exceptional case)? That would let it be easily installable via nimble at the cost of a bit more package size. Alternately, it could pull down a tarball of the correct source from github at install time.

[PMunch]

@pjz because when installing NimLSP most users already have the Nim sources on their machine, so why download a new copy or write a bunch of logic to pull down tarballs for something that's already on disk? It seems like maybe NimLSP has some trouble building on Mac and Windows because of not finding std/private/decode_helpers (see: #88). Normally building manually should just be a matter of passing -d:explicitSourcePath:<path to Nim sources> and it should build fine. Not entirely sure what this issue stems from though, so it might be hard to fix.

[pjz]

when installing NimLSP most users already have the Nim sources on their machine

I think this is incorrect. People who are new or trying out Nim (which are the people who need an LSP most!) probably got it installed via a package manager (apt, yum, whatever) and so do NOT have the Nim source on their machine.

[PMunch]

Depends on the package manager, the Arch package at least contains the sources. In fact the only ones I've heard who've had problems with Nim from a package manager in the past are Mac users. But I can see some newcomer installing Nim with an oddly packaged package and not having them only to be annoyed that installing NimLSP doesn't work. I guess NimLSP could do some checks to see if it can find the sources it needs (instead of just assuming that they're there and failing at the compilation stage if they are not) in order to spit out an informative message that you either need to point it at the correct sources or if you don't have them grab them from some URL.

[pjz]

...or make it clear that if you're packaging Nim, you need to package the source as well.

[lbartoletti]

Nim from a package manager in the past are Mac users.

I have the same problem on FreeBSD:

 Building nimlsp/nimlsp using c backend
/tmp/nimble_14015/githubcom_PMunchnimlsp/src/nimlsppkg/suggestlib.nim(11, 8) template/generic instantiation of `mImport` from here
/tmp/nimble_14015/githubcom_PMunchnimlsp/src/nimlsppkg/suggestlib.nim(7, 14) Error: cannot open file: /usr/local/nimsuggest/nimsuggest.nim

, and I guess on all system where nimsuggest.nim file is not installed. For example, on FreeBSD, nim files are installed in /usr/local/lib/nim It's possible to install nimsuggest/nimsuggest.nim into this directory, but as mentioned by @pjz "if you're packaging Nim, [make it clear that] you need to package the source as well."

[avently]

@PMunch

Depends on the package manager, the Arch package at least contains the sources

I'm on Manjaro (Arch), have Nim installed via pacman -S nim, sources are in /usr/lib/nim. There is no nimsuggest.nim there (/usr/bin/nimsuggest is present), only /usr/lib/nim/compiler/suggest.nim.

When I try to install nimlsp via nimble like so:

nimble install -d:explicitSourcePath=/usr/lib/nim/compiler nimlsp
Downloading https://github.com/PMunch/nimlsp using git
  Verifying dependencies for nimlsp@0.4.0
      Info: Dependency on jsonschema@>= 0.2.1 already satisfied
  Verifying dependencies for jsonschema@0.2.1
      Info: Dependency on ast_pattern_matching@any version already satisfied
  Verifying dependencies for ast_pattern_matching@1.0.0

And nothing more.
Tried like this:

nimble install -p:d:explicitSourcePath=/usr/lib/nim/compiler nimlsp
Downloading https://github.com/PMunch/nimlsp using git
  Verifying dependencies for nimlsp@0.4.0
      Info: Dependency on jsonschema@>= 0.2.1 already satisfied
  Verifying dependencies for jsonschema@0.2.1
      Info: Dependency on ast_pattern_matching@any version already satisfied
  Verifying dependencies for ast_pattern_matching@1.0.0
 Installing nimlsp@0.4.0
   Building nimlsp/nimlsp using c backend
Traceback from system (most recent call last)
Error: unhandled exception: No such file or directory
Additional info: "d:explicitSourcePath=/usr/lib/nim" [OSError]
    Prompt: Build failed for 'nimlsp@0.4.0', would you like to try installing 'nimlsp@#head' (latest unstable)? [y/N]
    Answer: ^CSIGINT: Interrupted by Ctrl-C.

Or like this:

nimble install -p:d:explicitSourcePath=/usr/lib/nim nimlsp
Downloading https://github.com/PMunch/nimlsp using git
  Verifying dependencies for nimlsp@0.4.0
      Info: Dependency on jsonschema@>= 0.2.1 already satisfied
  Verifying dependencies for jsonschema@0.2.1
      Info: Dependency on ast_pattern_matching@any version already satisfied
  Verifying dependencies for ast_pattern_matching@1.0.0
 Installing nimlsp@0.4.0
   Building nimlsp/nimlsp using c backend
Traceback from system (most recent call last)
Error: unhandled exception: No such file or directory
Additional info: "d:explicitSourcePath=/usr/lib/nim" [OSError]
    Prompt: Build failed for 'nimlsp@0.4.0', would you like to try installing 'nimlsp@#head' (latest unstable)? [y/N]
    Answer:

So, who to install Nimlsp then? I want to continue using Nim from Manjaro repos instead of choosenim.

[veksha]

why not use import nimsuggest/nimsuggest instead of mImport(explicitSourcePath / "nimsuggest" / "nimsuggest.nim") in suggestlib.nim file?
nimlsp shows a lot of errors on this file until i replace mImport line with import nimsuggest/nimsuggest

OS: Windows

i see that getCurrentCompilerExe() works correctly when you compile nimlsp with nim, but when you want to check its sources for errors with nimlsp - getCurrentCompilerExe() returns wrong path (a path not to nim sources, but to current project dir)

(after this change all errors in the file are gone and it still compiles correctly)

[SolitudeSF]

why not use import nimsuggest/nimsuggest instead of mImport(explicitSourcePath / "nimsuggest" / "nimsuggest.nim") in suggestlib.nim file?

its weird that this logic is replicated in code, when we have same almost same thing going on in config. i have a quick pr.

i see that getCurrentCompilerExe() works correctly when you compile nimlsp with nim, but when you want to check its sources for errors with nimlsp - getCurrentCompilerExe() returns wrong path (a path not to nim sources, but to current project dir)

you can import nimsuggest because its path is retrieved with getCurrentCompilerExe(), just in config, not in the importing module.

[SolitudeSF]

@veksha can you check if #124 works for you?

[veksha]

@veksha can you check if #124 works for you?

it works.
fom now on i see no lsp errors in suggestlib.nim file and still everything compiles with nimble or nim as before. (i had no troubles with compilation, only lsp errors)

[veksha]

@veksha can you check if #124 works for you?

@SolitudeSF , do you think it will compile for all OS'es? i mean import nimsuggest/nimsuggest

[SolitudeSF]

dont see why it would be os dependant

[cacilhas]

Still not working after applying #124:

./nimlsp/src/nimlsppkg/suggestlib.nim(4, 19) Error: cannot open file: nimsuggest/nimsuggest
       Tip: 5 messages have been suppressed, use --verbose to show them.
     Error: Build failed for package: nimlsp
        ... Execution failed with exit code 1
        ... Command: /bin/nim c --colors:on --noNimblePath -d:NimblePkgVersion=0.4.0 --path:~/.nimble/pkgs/jsonschema-0.2.1 --path:~/.nimble/pkgs/ast_pattern_matching-1.0.0 --hints:off -o:./nimlsp/nimlsp ./nimlsp/src/nimlsp.nim

What I’m doing wrong?

[SolitudeSF]

do you have nimsuggest source installed with nim compiler? that pr isnt meant to fix this issue (because the issue is in your setup)

[veksha]

Still not working after applying #124:

./nimlsp/src/nimlsppkg/suggestlib.nim(4, 19) Error: cannot open file: nimsuggest/nimsuggest
       Tip: 5 messages have been suppressed, use --verbose to show them.
     Error: Build failed for package: nimlsp
        ... Execution failed with exit code 1
        ... Command: /bin/nim c --colors:on --noNimblePath -d:NimblePkgVersion=0.4.0 --path:~/.nimble/pkgs/jsonschema-0.2.1 --path:~/.nimble/pkgs/ast_pattern_matching-1.0.0 --hints:off -o:./nimlsp/nimlsp ./nimlsp/src/nimlsp.nim

What I’m doing wrong?

that patch cleaned up nimlsp analysis errors for suggestlib.nim file and has nothing to do with compilation step of nimlsp.
you still need to pass -d:explicitSourcePath=... parameter.

[bung87]

can we make nim install script include nimsuggest and iterate these dirs https://github.com/nim-lang/Nim/blob/devel/tools/niminst/install.nimf

or download nimsuggest directory from git repo and import it?

[lim6112j]

Why should we download the sources for trying new language? I have dozens language installed on emacs, but this case is the first. I understand why it should be , but still stupid.

[PMunch]

It's a bit unfortunate. Granted you don't have to install NimLSP to try Nim, I used the language for years before I even wrote this project. Certainly nice to have, but in my mind not really a requisite. That said I guess we could set up a CD pipeline here on GitHub to build the binaries and simply distribute them. After all that's how you get Nim installed without sources in the first place.

[jinwoo]

The problem with the command nimble install -d:explicitSourcePath=<path> nimlsp is that the -d flag of nimble isn't what you think it is. It simply means "install dependencies only", so it stops after installing all the dependencies without installing nimlsp.

I'm not sure how I can make nimble pass forward that flag to the compiler.

[SolitudeSF]

Move the flag after the package name

[jinwoo]

Move the flag after the package name

@SolitudeSF Nope. Doesn't work. It still installs the dependencies only.

I installed nimlangserver instead. It seems to work fine with my Emacs.

[PMunch]

Seems like Nimble doesn't handle its arguments properly and confuses the shorthand -d with its own -d flag. Using the long version works though: nimble install nimlsp --define:explicitSourcePath=<path>.