chore(nix): cleanup

[Jint-lzxy]

This commit introduced three changes:

1. Opt for a shallow clone whenever we run the update_flake workflow since there's no need to retrive past commits.
2. Deleted result from .gitignore bc I couldn't track down its source (correct me if I'm wrong!)
3. Ensure either XCode or the CLT is installed on macOS before running otool. These aren't included by default on a fresh macOS install, and using xcrun can sort out those dependency snags.

btw @CharlesChiuGit @misumisumi do u think it's worth considering enforcing library policies for the pre-built binaries that Mason uses on macOS, similar to our approach on NixOS? This enforcement could involve setting DYLD_*_LIBRARY_PATH or utilizing install_name_tool (see manual below). One potential concern I see with this change is that macOS often includes software that hasn't been updated in a long time, so transitioning to a new version could potentially cause disruptions.

man install_name_tool

INSTALL_NAME_TOOL(1)                     General Commands Manual                     INSTALL_NAME_TOOL(1)

NAME
       install_name_tool - change dynamic shared library install names

SYNOPSIS
       install_name_tool [-change old new ] ... [-rpath old new ] ... [-add_rpath new ] ...
       [-delete_rpath new ] ... [-id name] file

DESCRIPTION
       Install_name_tool changes the dynamic shared library install names and or adds, changes or deletes
       the rpaths recorded in a Mach-O binary.  For this tool to work when the install names or rpaths
       are larger the binary should be built with the ld(1) -headerpad_max_install_names option.

       -change old new
              Changes the dependent shared library install name old to new in the specified Mach-O
              binary.  More than one of these options can be specified.  If the Mach-O binary does not
              contain the old install name in a specified -change option the option is ignored.

       -id name
              Changes the shared library identification name of a dynamic shared library to name.  If the
              Mach-O binary is not a dynamic shared library and the -id option is specified it is
              ignored.

       -rpath old new
              Changes the rpath path name old to new in the specified Mach-O binary.  More than one of
              these options can be specified.  If the Mach-O binary does not contain the old rpath path
              name in a specified -rpath it is an error.

       -add_rpath new
              Adds the rpath path name new in the specified Mach-O binary.  More than one of these
              options can be specified.  If the Mach-O binary already contains the new rpath path name
              specified in -add_rpath it is an error.

       -delete_rpath old
              deletes the rpath path name old in the specified Mach-O binary.  More than one of these
              options can be specified.  If the Mach-O binary does not contains the old rpath path name
              specified in -delete_rpath it is an error.

SEE ALSO
       dyld(1), ld(1), otool(1)

Apple, Inc.                                   March 4, 2009                          INSTALL_NAME_TOOL(1)

man dyld

DYLD(1)                                  General Commands Manual                                  DYLD(1)

NAME
       dyld - the dynamic linker

SYNOPSIS
       DYLD_FRAMEWORK_PATH
       DYLD_FALLBACK_FRAMEWORK_PATH
       DYLD_VERSIONED_FRAMEWORK_PATH
       DYLD_LIBRARY_PATH
       DYLD_FALLBACK_LIBRARY_PATH
       DYLD_VERSIONED_LIBRARY_PATH
       DYLD_IMAGE_SUFFIX
       DYLD_INSERT_LIBRARIES
       DYLD_PRINT_TO_FILE
       DYLD_PRINT_LIBRARIES
       DYLD_PRINT_LOADERS
       DYLD_PRINT_SEARCHING
       DYLD_PRINT_APIS
       DYLD_PRINT_BINDINGS
       DYLD_PRINT_INITIALIZERS
       DYLD_PRINT_SEGMENTS
       DYLD_PRINT_ENV
       DYLD_SHARED_REGION
       DYLD_SHARED_CACHE_DIR

DESCRIPTION
       The dynamic linker (dyld) checks the following environment variables during the launch of each
       process.
       Note: If System Integrity Protection is enabled, these environment variables are ignored when
       executing binaries protected by System Integrity Protection.

       DYLD_FRAMEWORK_PATH
              This is a colon separated list of directories that contain frameworks.  The dynamic linker
              searches these directories before it searches for the framework by its install name.  It
              allows you to test new versions of existing frameworks. (A framework is a library install
              name that ends in the form XXX.framework/Versions/A/XXX or XXX.framework/XXX, where XXX and
              A are any name.)

              For each framework that a program uses, the dynamic linker looks for the framework in each
              directory in DYLD_FRAMEWORK_PATH in turn. If it looks in all those directories and can't
              find the framework, it uses whatever it would have loaded if DYLD_FRAMEWORK_PATH had not
              been set.

              Use the -L option to otool(1) to discover the frameworks and shared libraries that the
              executable is linked against.

       DYLD_FALLBACK_FRAMEWORK_PATH
              This is a colon separated list of directories that contain frameworks.  If a framework is
              not found at its install path, dyld uses this as a list of directories to search for the
              framework.

              For new binaries (Fall 2023 or later) there is no default fallback.  For older binaries,
              their is a default fallback search path of: /Library/Frameworks:/System/Library/Frameworks

       DYLD_VERSIONED_FRAMEWORK_PATH
              This is a colon separated list of directories that contain potential override frameworks.
              The dynamic linker searches these directories for frameworks.  For each framework found
              dyld looks at its LC_ID_DYLIB and gets the current_version and install name.  Dyld then
              looks for the framework at the install name path.  Whichever has the larger current_version
              value will be used in the process whenever a framework with that install name is required.
              This is similar to DYLD_FRAMEWORK_PATH except instead of always overriding, it only
              overrides if the supplied framework is newer.  Note: dyld does not check the framework's
              Info.plist to find its version.  Dyld only checks the -current_version number supplied when
              the framework was created.

       DYLD_LIBRARY_PATH
              This is a colon separated list of directories that contain libraries. The dynamic linker
              searches these directories before it searches the default locations for libraries. It
              allows you to test new versions of existing libraries.

              For each dylib that a program uses, the dynamic linker looks for its leaf name in each
              directory in DYLD_LIBRARY_PATH.

              Use the -L option to otool(1) to discover the frameworks and shared libraries that the
              executable is linked against.

       DYLD_FALLBACK_LIBRARY_PATH
              This is a colon separated list of directories that contain libraries.  If a dylib is not
              found at its install  path, dyld uses this as a list of directories to search for the
              dylib.

              For new binaries (Fall 2023 or later) there is no default.  For older binaries, their is a
              default fallback search path of: /usr/local/lib:/usr/lib.

       DYLD_VERSIONED_LIBRARY_PATH
              This is a colon separated list of directories that contain potential override libraries.
              The dynamic linker searches these directories for dynamic libraries.  For each library
              found dyld looks at its LC_ID_DYLIB and gets the current_version and install name.  Dyld
              then looks for the library at the install name path.  Whichever has the larger
              current_version value will be used in the process whenever a dylib with that install name
              is required.  This is similar to DYLD_LIBRARY_PATH except instead of always overriding, it
              only overrides is the supplied library is newer.

       DYLD_IMAGE_SUFFIX
              This is set to a string of a suffix to try to be used for all shared libraries used by the
              program.  For libraries ending in ".dylib" the suffix is applied just before the ".dylib".
              For all other libraries the suffix is appended to the library name.  This is useful for
              using conventional "_profile" and "_debug" libraries and frameworks.

       DYLD_INSERT_LIBRARIES
              This is a colon separated list of additional dynamic libraries to load before the ones
              specified in the program. If instead, your goal is to substitute a library that would
              normally be loaded, use DYLD_LIBRARY_PATH or DYLD_FRAMEWORK_PATH instead.

       DYLD_PRINT_TO_FILE
              This is a path to a (writable) file. Normally, the dynamic linker writes all logging output
              (triggered by DYLD_PRINT_* settings) to file descriptor 2 (which is usually stderr).  But
              this setting causes the dynamic linker to write logging output to the specified file.

       DYLD_PRINT_ENV
              If set, causes dyld to print a line of key=valule for each enviroment variable in the
              process.

       DYLD_PRINT_LIBRARIES
              If set, causes dyld to print a line for each mach-o image loaded into a process.  This is
              useful to make sure that the use of DYLD_LIBRARY_PATH is getting what you want.

       DYLD_PRINT_LOADERS
              If set, causes dyld to print a line whether each image is tracked by a JustInTimeLoader or
              a PrebuiltLoader.  Additionally, it prints if a PrebuiltLoaderSet was used to launch the
              process or if a PrebuiltLoader was written to make the next launch faster.

       DYLD_PRINT_SEARCHING
              If set, causes dyld to print a line about each file system path checked when searching for
              an image to load.

       DYLD_PRINT_INITIALIZERS
              If set, causes dyld to print out a line when running each initializer in every image.
              Initializers run by dyld include constructors for C++ statically allocated objects,
              functions marked with __attribute__((constructor)), and -init functions.

       DYLD_PRINT_APIS
              If set, causes dyld to print a line whenever a dyld API is called (e.g. dlopen()).

       DYLD_PRINT_SEGMENTS
              If set, causes dyld to print out a line containing the name and address range of each mach-
              o segment that dyld maps.  In addition it prints information about if the image was from
              the dyld shared cache.

       DYLD_PRINT_BINDINGS
              If set, causes dyld to print a line each time a symbolic name is bound.

       DYLD_SHARED_REGION
              This can be "use" (the default) or "private".  Setting it to "private" tells dyld to remove
              the shared region from the process address space and mmap() back in a private copy of the
              dyld shared cache in the shared region address range. This is only useful if the shared
              cache on disk has been updated and is different than the shared cache in use.

       DYLD_SHARED_CACHE_DIR
              This is a directory containing dyld shared cache files.  This variable can be used in
              conjunction with DYLD_SHARED_REGION=private to run a process with an alternate shared
              cache.

DYNAMIC LIBRARY LOADING
       Unlike many other operating systems, Darwin does not locate dependent dynamic libraries via their
       leaf file name.  Instead the full path to each dylib is used (e.g. /usr/lib/libSystem.B.dylib).
       But there are times when a full path is not appropriate; for instance, may want your binaries to
       be installable in anywhere on the disk.  To support that, there are three @xxx/ variables that can
       be used as a path prefix.  At runtime dyld substitutes a dynamically generated path for the @xxx/
       prefix.

       @executable_path/
              This variable is replaced with the path to the directory containing the main executable for
              the process.  This is useful for loading dylibs/frameworks embedded in a .app directory.
              If the main executable file is at /some/path/My.app/Contents/MacOS/My and a framework dylib
              file is at /some/path/My.app/Contents/Frameworks/Foo.framework/Versions/A/Foo, then the
              framework load path could be encoded as
              @executable_path/../Frameworks/Foo.framework/Versions/A/Foo and the .app directory could be
              moved around in the file system and dyld will still be able to load the embedded framework.

       @loader_path/
              This variable is replaced with the path to the directory containing the mach-o binary which
              contains the load command using @loader_path. Thus, in every binary, @loader_path resolves
              to a different path, whereas @executable_path always resolves to the same path.
              @loader_path is useful as the load path for a framework/dylib embedded in a plug-in, if the
              final file system location of the plugin-in unknown (so absolute paths cannot be used) or
              if the plug-in is used by multiple applications (so @executable_path cannot be used). If
              the plug-in mach-o file is at /some/path/Myfilter.plugin/Contents/MacOS/Myfilter and a
              framework dylib file is at
              /some/path/Myfilter.plugin/Contents/Frameworks/Foo.framework/Versions/A/Foo, then the
              framework load path could be encoded as
              @loader_path/../Frameworks/Foo.framework/Versions/A/Foo and the Myfilter.plugin directory
              could be moved around in the file system and dyld will still be able to load the embedded
              framework.

       @rpath/
              Dyld maintains a current stack of paths called the run path list.  When @rpath is
              encountered it is substituted with each path in the run path list until a loadable dylib if
              found.  The run path stack is built from the LC_RPATH load commands in the depencency chain
              that lead to the current dylib load.  You can add an LC_RPATH load command to an image with
              the -rpath option to ld(1).  You can even add a LC_RPATH load command path that starts with
              @loader_path/, and it will push a path on the run path stack that relative to the image
              containing the LC_RPATH.  The use of @rpath is most useful when you have a complex
              directory structure of programs and dylibs which can be installed anywhere, but keep their
              relative positions.  This scenario could be implemented using @loader_path, but every
              client of a dylib could need a different load path because its relative position in the
              file system is different. The use of @rpath introduces a level of indirection that
              simplifies things.  You pick a location in your directory structure as an anchor point.
              Each dylib then gets an install path that starts with @rpath and is the path to the dylib
              relative to the anchor point. Each main executable is linked with -rpath @loader_path/zzz,
              where zzz is the path from the executable to the anchor point.  At runtime dyld sets it run
              path to be the anchor point, then each dylib is found relative to the anchor point.

SEE ALSO
       dyldinfo(1), ld(1), otool(1)

Apple Inc.                                     June 1, 2020                                       DYLD(1)

[misumisumi]

Deleted result from .gitignore bc I couldn't track down its source (correct me if I'm wrong!)

This was added to prevent accidentally committing this because when you run nix build without specifying --out-link, an artifact is generated in ./result.
You can delete this if you don't need it.

[misumisumi]

btw @CharlesChiuGit @misumisumi do u think it's worth considering enforcing library policies for the pre-built binaries that Mason uses on macOS, similar to our approach on NixOS?

Hmm.
I can't answer that question because I don't have a Mac environment at hand.
If it's a general question that doesn't depend on nixpkgs (such as when you installed dependencies with homebrew), I think it's a problem with mason.nvim or a symmetric package.

---

We can check if nixpkgs is linked with check-linker.
check-linker checks whether dependencies are linked to /nix/store in Nix environments, regardless of whether they are NixOS or non-NixOS.
Although it was not explained in #1250 , this script will scan $XDG_DATA_HOME/$NVIM_APPNAME/mason/bin under NVIM_APPNAME if it is specified, and $XDG_DATA_HOME/nvim/mason/bin if not.

[misumisumi]

This enforcement could involve setting DYLD_*_LIBRARY_PATH or utilizing install_name_tool (see manual below).

If we need DYLD_*_LIBRARY_PATH settings, it's easy to add them too.
We only add environment valiable or extraWrapperArgs in nixos/neovim/default.nix.

[CharlesChiuGit]

tbh, i just knew we can set DYLD_*_LIBRARY_PATH or utilizing install_name_tool, newbie me lol.
I think specifying DYLD_*_LIBRARY_PATH sounds like a gd idea to me, so we can make things more standardized.

One potential concern I see with this change is that macOS often includes software that hasn't been updated in a long time

i think this should be fine since those fundamental libs generally have gd backward compatibility?( correct me if i'm wrong lol

[Cyberczy]

@misumisumi Can u tell me how to create a package using Nix, or what should I learn?

[misumisumi]

You can learn from official nixpkg manual.
There are also articles written by me, although they are in Japanese.
You can learn further understanding and application by referring to the actual package of NixOS/nixpkgs.

[Jint-lzxy]

@misumisumi @CharlesChiuGit Apologies for the delayed response and the oversight in explaining the issue! Let me provide further detail on how shared libraries are located on macOS (Darwin). Unlike many other Unix-like operating systems, Darwin's dynamic linker, dyld, does not find dependent shared libraries based on their leaf filename (i.e., the one obtained using basename). Instead, it directly utilizes the paths hardcoded in the executable. For instance, consider these two examples (ld and dyld, with simplified output):

$ ldd a.out
	libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000fffff76c0000)

and

$ otool -L a.out
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

The crucial difference is that on Darwin, dyld only¹ tries to find libSystem.B.dylib at /usr/lib/libSystem.B.dylib. If it cannot find libSystem under /usr/lib/, it directly fails, displaying an error message like this:

dyld[48101]: Library not loaded: /usr/lib/libSystem.B.dylib
  Referenced from: <08197395-BB6E-33C2-BB73-C47F2C1C672E> /private/var/folders/jy/91xg_8j9169dydct2k4r7yw80000gn/T/a.out.cdBaJwRHju/a.out
  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libSystem.B.dylib' (no such file), '/usr/lib/libSystem.B.dylib' (no such file, not in dyld cache)
[1]    48101 abort      ./a.out

Which is why I mentioned earlier in #1252 (comment) that on Darwin, we may only² either use install_name_tool to change the hardcoded path on Mason's pre-built executables³ to point to the Nix store, or specify the DYLD_*_LIBRARY_PATH environment variable to make dyld operate akin to ld, dynamically locating libraries during runtime with specified paths. Hopefully this clears up some confusion regarding shared libraries on macOS :D

Footnotes

1. Actually dyld would also search for libSystem.B.dylib inside its shared cache using symbol stubs, and scrutinize several MachO-specific load commands like LC_RPATH in an attempt to find it, but these complexities are all unnecessary here. ↩

2. Package managers following the "traditional" directory hierarchy are usually considered, with their paths automatically included in the executables' @rpath. These paths are prioritized first where applicable, such as /usr/local/lib/ for Homebrew formulas. ↩

3. Since they're not utilizing our dependencies installed with Nix, imo it diminishes the value of using Nix on macOS. ↩

[Jint-lzxy]

i think this should be fine since those fundamental libs generally have gd backward compatibility?( correct me if i'm wrong lol

@CharlesChiuGit Yeah u're absolutely right lol. Actually, what I meant to say was that those "new" features that come with some packages could alter the way Neovim operates. For instance, ncurses 6 introduces a bunch of new features (and databases) that ncurses 5 doesn't have. So, if we now instruct some executable to use ncurses 6 (macOS ships with a really ancient version of ncurses 5), the display could look different (like with colors).

[CharlesChiuGit]

so set DYLD_*_LIBRARY_PATH to nix's lib should make things more compatible w/ new features?
for example, set DYLD_*_LIBRARY_PATH to .local/state/nix/profile/lib(xdg = true) ,.nixprofile/lib(xdg = false) or sth similar like

nvimdots/nixos/neovim/default.nix

Lines 106 to 111 in e3e5797

	''CPATH=''${CPATH:+''${CPATH}:}${neovim-build-deps}/include''
	''CPLUS_INCLUDE_PATH=''${CPLUS_INCLUDE_PATH:+''${CPLUS_INCLUDE_PATH}:}:${neovim-build-deps}/include/c++/v1''
	''LD_LIBRARY_PATH=''${LD_LIBRARY_PATH:+''${LD_LIBRARY_PATH}:}${neovim-build-deps}/lib''
	''LIBRARY_PATH=''${LIBRARY_PATH:+''${LIBRARY_PATH}:}${neovim-build-deps}/lib''
	''NIX_LD_LIBRARY_PATH=''${NIX_LD_LIBRARY_PATH:+''${NIX_LD_LIBRARY_PATH}:}${neovim-build-deps}/lib''
	''PKG_CONFIG_PATH=''${PKG_CONFIG_PATH:+''${PKG_CONFIG_PATH}:}${neovim-build-deps}/include/pkgconfig''

?

[Jint-lzxy]

so set DYLD_*_LIBRARY_PATH to nix's lib should make things more compatible w/ new features?

Ye and IMO this should be the most straightforward method. However, the drawback here is that other processes spawned by Neovim (e.g., lazygit) that aren't managed by Nix could also be impacted during library resolution. On the other hand, using install_name_tool is more intricate but carries fewer side effects.

[misumisumi]

This discussion is similar to dealing with binary packages in NixOS.
When using binary on NixOS, there is a way to rewrite the path to dylib using patchelf. (This is other method adopted here., more details Packaging/Binaries)
I previously used a method to automatically rewrite dylib using mason.nvim's hook.
I settled on the current method because it was incomplete in various ways, but it should be a start for discussion it.
More details, under article (This is written by Japanese.)
NixOSでmason.nvimで入れたLSP serverを使えるようにする

[misumisumi]

However, the drawback here is that other processes spawned by Neovim (e.g., lazygit) that aren't managed by Nix could also be impacted during library resolution.

That's true.
In addition, quickfix and command execution in the internal terminal are affected.
However, I decided that this problem could be avoided by using a multiplexer, so I adopted the current method.

[misumisumi]

On the other hand, using install_name_tool is more intricate but carries fewer side effects.

IMO, if we use this method to solve the problem on macOS, we should be able to use almost the same method on NixOS, so it is better to replace both.

[CharlesChiuGit]

just a side note, i just found out that upgrading macOS will break nix, since it regenerate /etc/zshrc, but it's also easy to fix. see the fix here.

maybe jint will experiences similar issues lol

[Jint-lzxy]

just a side note, i just found out that upgrading macOS will break nix, since it regenerate /etc/zshrc, but it's also easy to fix.

@CharlesChiuGit I just spotted this in my inbox and wanted to quickly mention that I managed to "circumvent" the issue by setting the system immutable flag on /etc/zshrc:

sudo chflags simmutable /etc/zshrc

bc I discovered that the file's content hasn't changed since Catalina (the first version to use zsh as the default shell). AFAIU this tweak ensures the installer tool (the underlying exec for system software updates) can't undo my changes with each update. But to make any modifications, the file has to be temporarily "unlocked" using:

sudo chflags nosimmutable /etc/zshrc

See chflags(1), installer(8).

[CharlesChiuGit]

damn, i completely brake my nix environment and can't get it fixed 🥲 (reinstalled few times lol

ok, i finally got it fixed lol
i found this interesting repo that claims to prevent nix to break when doing OS upgrade, testing it now.
https://github.com/determinatesystems/nix-installer

[misumisumi]

The section on nixos in the wiki is out of date and needs to be updated
Also, lazy.nvim is not designed to be a read-only file, so the current arrangement will update the file, but it will give an error.
Should we fix this?
I will be busy until mid-June, so I may not be able to respond much

[Jint-lzxy]

I'm still new to Nix(OS) so plz correct me if any of the following is conceptually wrong!

I settled on the current method because it was incomplete in various ways, but it should be a start for discussion it.
...
IMO, if we use this method to solve the problem on macOS, we should be able to use almost the same method on NixOS, so it is better to replace both.

So iiuc depending on how much extra work is needed, here are our options:

1. Utilize environment variables to help the linker locate the necessary dependencies.
   * The drawback here is that it impacts every subprocess launched by Neovim, and there's no simple workaround. However, I'm not entirely convinced this is a critical concern.
2. Skip the entire package system (aka Mason) and utilize the executables provided by the native package manager instead.
   * The downside is we might have to enhance the current LSP "framework" significantly to handle locally installed executables first (and bypass Mason when necessary). Also, not all Mason packages might be available through native package managers, and some users might have a messy local storage, leading to unexpected issues.
3. Modify the executables directly using the appropriate tools to ensure permanent changes.
   * The downside is that it could be challenging to reliably parse the dynamic linker's output and then systematically feed it to another program (e.g., nix path-info).

Honestly, I actually lean towards option 2 bc it's the most stable one and also tries to be as Nix-ish as possible. It can accommodate existing executables and also conserve disk space. What do u guys think?

The section on nixos in the wiki is out of date and needs to be updated

I suggest postponing this until we reach a consensus on the implementation. IMO it's probable that our implementation will change in the future, and documenting those volatile configs doesn't sit well with anyone at all.

Also, lazy.nvim is not designed to be a read-only file, so the current arrangement will update the file, but it will give an error.

I think if we're dead set on making the entire config immutable, we could have GitHub actions handle the updates. Then, on the end user's local setup, they could simply do a :Lazy restore (as opposed to :Lazy sync/update) to stay in sync with the changes. This approach won't mess with the lockfile and could still guarantee periodic lockfile updates (which we could control using an actions cronjob).

Alternatively, if we're not too strict about restricted write access, we could stash the lockfile in a separate location¹ like stdpath("state")/lazy-lock.json, keeping it apart from our config. This way, the existing config remains untouched, and all we have to do is tweak the install script to reroute the lockfile in our repository to this designated location for machines adhering to the traditional filesystem hierarchy.

Footnotes

1. Specifically, by setting lockfile = vim.fn.stdpath("state") .. "/lazy-lock.json". ↩

[misumisumi]

Honestly, I actually lean towards option 2 bc it's the most stable one and also tries to be as Nix-ish as possible.

I also think this is the best.
However, as you say, major changes are required around the LSP configuration.
This is because mason-lsp-config cannot be used.

---

IMO, is better creating plugin for bypass when exsting global pkg.
Also valid for apt, pacman, homebrew (mac), scoop (win).
If it has already been installed globally using the mason.nvim API, it would be nice if it could be bypassed as installed.
Lack of experience and time to create plugins.
Other option, configuration native lsp. (without mason ecosystem.)

[misumisumi]

they could simply do a :Lazy restore

I recently learned this.
Maybe you don't need to worry too much about the lazy.nvim error.