Changes to the mason.nvim registry

[williamboman]

Hello! This is a PSA for people who use mason.nvim (tl;dr at the bottom). Going forward the core registry will be hosted as a collection of static, YAML-based, package definitions at https://github.com/mason-org/mason-registry, as opposed to bundled with the mason.nvim plugin itself as Lua modules. This decouples the infrastructure in such a way that the Lua plugin becomes a standalone client that can be hooked into more registries than just the core one, as well as more easily be subject to versioning (mason.nvim currently doesn't version itself due to the fact that packages are bundled with it, which complicates things). This decoupling also lowers the barrier of entry for additional maintainers who can solely focus on the registry, without having to learn the internals of the plugin (more info will come soon if you're interested in possibly helping maintain the core registry).

The migration will be entirely seamless, the plan is to simply replace the Lua-based packages in batches.

For those interested, some notable features of the new registry specification are:

• Makes heavy use of purls to define package sources. This aids portability of package identifiers, which is currently leveraged to automate version upgrades through Renovate and hopefully can be used to hook into vulnerability databases such as NVD for automated security scanning purposes.
• All package installations are now tested across all supported platforms (Linux, macOS, Windows) via GitHub runners. This is huge from a stability and maintenance perspective as I've put a lot of "invisible" effort into testing these manually so far.
• Links more files than just executables. Two new types of link targets have been added, for "share" and "opt" files. This makes it possible to provide a stable location for non-executable files. One example of such a use case are .jar files, which can be accessed via vim.fn.expand("$MASON/share/some-package/the.jar") instead of reaching into internal package structures (which is unsupported and prone to breakage across versions).
• Package definitions are well-defined and strictly validated against a JSON schema. This helps tremendously with catching inconsistencies early (which is something I've had to do manually so far) but also with editor tooling such as diagnostics and autocompletion via the YAML language server.
• Has support for declaring installation instructions for older versions of a package (instructions that no longer are valid for later versions).
• Includes SPDX license identifiers for each package.

With the stability that this new registry brings I will start looking into some old, popular, feature requests such as a locking mechanism and automatic installation.

I also want to explore adding treesitter and plugin support. The latter should be fairly straight forward as the infrastructure is largely in place already, but is something I've deprioritized due to the introduction of the excellent lazy.nvim plugin manager.

I also want to thank current & past GitHub sponsors who help finance costs associated with the plugin. I regularly pay the surplus forward to other devs whose tooling I heavily rely on (huge shout-out to sumneko for working on the Lua language server, without it a plugin of the complexity of mason.nvim would be impossible, go sponsor them here).

tl;dr: the core mason.nvim registry is moving to https://github.com/mason-org/mason-registry. this will be a fully seamless migration so you shouldn't notice anything.

[baggiponte]

Congratulations for the effort and the communication! What does this mean for plugins such as mason-lspconfig, mason-dap...?

[williamboman]

Thanks! It doesn't really mean much at all. This is just migrating the core registry from one place to another. However, in doing so, support for customizing registries was added to mason.nvim. By default it uses the core registries which contain the packages that those extension plugins rely on. Should a user override the registries configuration, and in doing so exclude the core registries, the plugins you mention would certainly struggle to function properly. The registries configuration is labeled as an "Advanced setting", and I don't believe any user would inadvertently exclude the core registry.

tl;dr: unless explicitly configured, those plugins are unaffected

[williamboman]

Also, if you're programmatically interfacing with mason.nvim via Lua APIs you might want to consider the scenario where the registry has not been downloaded yet (e.g. completely fresh installations). Previously one would not have to consider this because packages were bundled with the plugin itself. I don't expect this to be an issue, but it's worth pointing out.

There are two new functions added to the mason-registry API for this:

local registry = require("mason-registry")

-- --- snip ---

-- :h mason-registry.refresh()
registry.refresh(function ()
    -- This callback is called when the local registry is reasonably up to date.
    -- Interacting with the registry, such as getting packages, can be done with
    -- reasonable certainty that they can be loaded.
end)

-- You can also refresh the registry synchronously (i.e. blocks execution) if working with callbacks is difficult.
-- Note that this freezes the editor if the registry needs to be updated (otherwise it yields back immediately).
registry.refresh()

-- --- snip ---

-- :h mason-registry.update()
-- To "force" an update of the registry you can use the .update() function instead.
-- This can also be done via the :MasonUpdate command (which prints feedback to vim.notify).
registry.update(function (ok, err)
    if ok then
        -- registry was successfully updated
    else
        -- registry failed to update
    end
end)

You can also pin the registry version when setting up Mason (I don't recommend doing it but there are use cases for it), like so:

require("mason").setup {
    registries = {
        "lua:mason-registry.index", -- over time this registry will be depleted of packages and replaced with the one below

        -- Note that the mason-org/mason-registry very frequently releases new versions (usually many times a day).
        -- Pinning the version means that users wont receive new packages or version upgrades.
        "github:mason-org/mason-registry@2023-03-20-knotty-zipper" -- pinned version
    }
}

---

Ping @kylo252 @folke @mehalter @siduck et al. If you have a bootstrap step in your distributions you might want to consider incorporating :h mason-registry.update() in it. If you otherwise directly interact with Mason's Lua APIs to retrieve packages, you might want to consider what I've mentioned above. Mason does automatically download the registry as-needed, for example when installing via the command API (:MasonInstall) or when opening the Mason window (:Mason).

[siduck]

also @williamboman is this error normal? I get it with vim.cmd("MasonInstall " .. "lua-language-server stylua")

I noticed that lua-language-server listed in my table doesnt download , whereas other packages do download.

vid.mp4

[williamboman]

Have you called require("mason").setup() before interacting with mason-registry?

[williamboman]

Ah sorry I didn't watch the video until now. That should not be erroring, will fix!

[williamboman]

@siduck #1128 should fix this!

[siduck]

thanks man works now!

[folke]

So if I understand correctly, it would be good to add registry.refresh() to a plugin's build step? So that it would refresh the local registry when the mason plugin gets installed and updated?

Or is it better to just always execute the refresh(callback) code when loading mason-lspconfig for example?

Will the refresh callback then always be executed? Even when the registry didn't update?

Would mason-lspconfig's get_available_servers also initially return an empty list when the local registry doesn't exist yet? Or are the mappings still going to stay in the mason-lspconfig repo?

Would mason-lspconfig ensure_installed automatically download the registry when needed?

Sorry for all the questions :) Nice update btw!!

[williamboman]

So if I understand correctly, it would be good to add registry.refresh() to a plugin's build step? So that it would refresh the local registry when the mason plugin gets installed and updated?

It's a trade-off to be made imo, eager vs lazy ^🥁 download. I think doing it in connection with a plugin update makes sense, it doesn't require much resources at all, it's asynchronous, and I'd wager that p99 < 500ms is true, so deferring it has questionable benefits. You should always opt for the async variant (passing a callback) over the synchronous one though, for better UX. The synchronous variant exists as an escape hatch, for example when interfacing with synchronous Neovim APIs.

Or is it better to just always execute the refresh(callback) code when loading mason-lspconfig for example?

Good question! You shouldn't have to worry about the state of the registry on behalf of mason-lspconfig, it will ensure it interacts with the registry appropriately. Also, and I might be missing something, mason-lspconfig only really functions if you've installed packages already (i.e. you already have the necessary registries locally).

Would mason-lspconfig's get_available_servers also initially return an empty list when the local registry doesn't exist yet? Or are the mappings still going to stay in the mason-lspconfig repo?

That function currently sources candidates from registry.get_all_package_names() which it then checks against a hard-coded mapping, so it would return an empty list if there is no local registry. For now, if you want to ensure this doesn't happen I'd wrap any interactions with that function with registry.refresh(), like so:

registry.refresh(function ()
    require("mason-lspconfig").get_available_servers()
end)

Would mason-lspconfig ensure_installed automatically download the registry when needed?

Soon ™️, just struggling with some tests.

Sorry for all the questions :) Nice update btw!!

Good questions, and thanks!

Also just to clarify some slightly missing info - the core registry still resides within the mason.nvim plugin today, so nothing has changed since yesterday other than a 1-line change in the default settings table. The plan is to gradually "deplete" the Lua-based package index over time at which point the considerations mentioned in this thread start to materialize.

[folke]

ok, cool. Thank you for the info!

So it seems for LazyVim, I don't need to make any changes to the lsp code.
I have an ensure_installed for regular mason too though, but will wrap that in refresh if it exists.

[folke]

no wait, I'm wrong. I do need to fix the lsp code :)

[williamboman]

Will the refresh callback then always be executed? Even when the registry didn't update?

Missed this one. Yep it's always executed if registry didn't update (this is the most common outcome due to local cache TTL, it will yield back immediately most of the time) or even if it failed to update.

[folke]

LazyVim should be using the new registry now: LazyVim/LazyVim@df5cbfd

[kylo252]

Amazing work so far!

A bit late to the party, but I'm a bit uncertain about some a few things

• versioning: it is not obvious to me how the tag/commit of the registry is specified. I guess the user can switch it out completely, and be done with it?
• integrity: related to the previous point, but why not checkout the git repo directly?
• specs (implementation detail): why not use json directly instead of yaml?
• engine/back-end (implementation detail): have you considered relying on an already established package manager, to reduce further security and/or maintenance burdens? I see you're already implementing similar concepts to vcpkg, so why not use it directly? 👀

[williamboman]

versioning: it is not obvious to me how the tag/commit of the registry is specified. I guess the user can switch it out completely, and be done with it?

You can pin the registry version when setting up mason, like so:

require("mason").setup {
    registries = {
        "lua:mason-registry.index", -- over time this registry will be depleted of packages and replaced with the one below

        -- Note that the mason-org/mason-registry very frequently releases new versions (usually many times a day).
        -- Pinning the version means that users wont receive new packages or version upgrades.
        "github:mason-org/mason-registry@2023-03-20-knotty-zipper" -- pinned version
    }
}

I implemented it this way because I find the idea of decoupling it like this pretty appealing, one could repurpose the Mason client for a completely custom set up. It's probably an extraordinarily rare use case though.

integrity: related to the previous point, but why not checkout the git repo directly?

The registry uses YAML for package definitions and I was unable to find a nice compromise for parsing YAML files. I tried to find Lua-based YAML parsers but they were all very limited (and very poor in terms of performance), and I definitely don't want to write one myself. The solution I came up with was uploading JSON assets to the GitHub release pipeline. I'm not sure how it compares in terms of integrity though, would you mind elaborate?

specs (implementation detail): why not use json directly instead of yaml?

JSON is imho a horrible data format for this purpose and hinders maintainability (ugly diffs, too verbose).

engine/back-end (implementation detail): have you considered relying on an already established package manager, to reduce further security and/or maintenance burdens? I see you're already implementing similar concepts to vcpkg, so why not use it directly? 👀

Not too familiar with vcpkg, I'll check it out in more detail. The primary reason is portability, the Mason client runs entirely on the embedded Lua runtime, with no external dependencies (or, a few dependencies on core system utilities like curl, unzip, etc). My goal with Mason is to keep it entirely native to Neovim, so the added flexibility of having a proprietary implementation targeting Neovim only makes it very easy to steer the project in whatever direction makes sense.

In terms of maintenance burden, the core Mason infrastructure is more or less "done" in terms of the current feature scope. The majority of code traffic is in the registry, which is now being moved elsewhere. I don't predict there being much activity in the core infrastructure going forward except for some QoL features.

[kylo252]

[...] The solution I came up with was uploading JSON assets to the GitHub release pipeline. I'm not sure how it compares in terms of integrity though, would you mind elaborate?

I just meant the integrity of the registry itself, since you'd be able to use git diff with a repo. An alternative would be to store the file hash, or maybe always overwrite the local version from upstream (?)

Not too familiar with vcpkg, I'll check it out in more detail.

managing (pre-compiled) tools is a little bit under represented, but take a look at these examples:

• https://github.com/microsoft/vcpkg/tree/master/ports/vcpkg-tool-bazel
• https://github.com/microsoft/vcpkg/tree/master/ports/vcpkg-tool-gn

FWIW, vcpkg can bootstrap itself into a portable directory, so no additional runtime requirements are needed. If anything, it allows you to define requirements into the specs, e.g. nodejs v15+

I'm not sure it's a perfect fit tbh, but I can see a lot of similarities between what you're trying to do here and what comes for free with that ecosystem, including:

With the stability that this new registry brings I will start looking into some old, popular, feature requests such as a locking mechanism and automatic installation.

[williamboman]

I just meant the integrity of the registry itself, since you'd be able to use git diff with a repo. An alternative would be to store the file hash, or maybe always overwrite the local version from upstream (?)

Ah I see. So the registry provides a checksums.txt containing sha256 sums of the zipped as well as unzipped JSON file. These checksums are downloaded and stored in the info.json file, which is located adjacent to the registry.json file. This gives you the ability to at least verify that the contents of the registries hasn't changed since it was downloaded (and if this is something you have a need to check, then the contents of your local Mason registry is the least of your concerns 😅). At the point of download, integrity is essentially delegated to TLS & TCP.

I'm not sure it's a perfect fit tbh, but I can see a lot of similarities between what you're trying to do here and what comes for free with that ecosystem, including:

https://xkcd.com/927/

[WillEhrendreich]

would this help more advanced lsp setup scenarios like for fsharp, when fsautocomplete can be used on it's own, but if you want to use Ionide, the actual lsp should be downloaded and installed, but setup and loaded through Ionide? I think Java has something funky like that too right? and rust? Am I understanding correctly that it could have another registration that could be doing this automatically, or is my ignorance showing again? lol.

[williamboman]

Hey! Hmm I'm not sure I'm following. This just changes how the registry is defined and hosted, and doesn't really impact the feature set per se.

[WillEhrendreich]

ok, thanks, I'm obviously not understanding something then, haha.

[williamboman]

Language servers installed via Mason are not in any way constrained to the native LSP client (or nvim-lspconfig) in any way. One could pick any LSP client, such as https://github.com/prabirshrestha/vim-lsp, or a dedicated F# plugin that handles the server for you. Mason just installs it.

[surmish]

GitHubRegistrySource(repo=mason-org/mason-registry) failed to install: Failed to fetch latest registry version from GitHub API.

Is there a manual fix to get around this error?

[ewanm89]

Install github cli, make sure gh is in PATH and authenticate to githuib with it.

I would prefer if we just used raw git and didn't have to go via github only API and tools. Currently it is not clear gh cli is needed either, neither does checkhealth check for it.

[williamboman]

@ewanm89 That shouldn't be a requirement, seems like a small oversight. #1181 should make gh optional.

[ewanm89]

Oh, I hadn't dove fully into the code to fix it yet, I found another old reference to it in regards to using Mason with LunarVIM on their tracker and tried installing gh in the end, I'll gladly love to remove it again. :D

[Kuon-dev]

I went with this attempt fix, but i got a new error instead

GitHubRegistrySource(repo=mason-org/mason-registry) failed to install: Failed to download registry archive.

Any fix for this? Pressing C to install still shows the same error

[ewanm89]

Updating the plugin(s) should have you using curl/wget now, William updated it just fine and I have tested it here.

[ismet55555]

So I'm using AstroNvim configurations. It is using Lazy.nvim to handle packages, I just upgraded to their latest version and I can't seem to get over this error (as commented above). This works on my Ubuntu Linux x86_64 computer, however does not work on my MacOS M1 ARM (although don't think architecture matters here)

When I :MasonLog I get lots of this:

...
[ERROR Mon Apr 24 14:10:20 2023] ...im/lazy/mason.nvim/lua/mason-registry/sources/github.lua:181: Failed to install registry GitHubRegistrySource(repo=mason-org/mason-registry). "Failed to download registry archive."
[ERROR Mon Apr 24 14:10:28 2023] ...l/share/nvim/lazy/mason.nvim/lua/mason-registry/init.lua:79: Cannot find package "\"python\"".
[ERROR Mon Apr 24 14:10:28 2023] ...l/share/nvim/lazy/mason.nvim/lua/mason-registry/init.lua:79: Cannot find package "\"bash-debug-adapter\"".
[ERROR Mon Apr 24 14:10:28 2023] ...l/share/nvim/lazy/mason.nvim/lua/mason-registry/init.lua:79: Cannot find package "\"debugpy\"".
[ERROR Mon Apr 24 14:13:21 2023] ...l/share/nvim/lazy/mason.nvim/lua/mason-registry/init.lua:79: Cannot find package "\"pyright\"".
...

I added the following to my mason.lua config file:

  {
   "williamboman/mason.nvim",
   opts = {
    registries = {
     "lua:mason-registry.index"
    },
    PATH = "append"
   }
  },

This effectively gets rid of the error message, however, it shows no listed packages/modules, everything is zeros. In :MasonLogs I get the same logs.

I'm not 100% sure if this is related but it is very frustrating!

[mrthere3]

I also get the same problem

[rdataforge]

same issue here in windows 10 WSL. any hint?

[extde]

I have same issue:
GitHubRegistrySource(repo=mason-org/mason-registry) failed to install: Failed to fetch latest registry version from GitHub API.
Fixed with installing curl. But then still got next issue:
GitHubRegistrySource(repo=mason-org/mason-registry) failed to install: Failed to download registry archive.

After looking at logs I found that wget is behaving incorrectly.
[ERROR Sat Jan 20 17:13:31 2024] ...e/nvim/lazy/mason.nvim/lua/mason-core/providers/init.lua:80: Provider "github" "get_latest_release" failed: spawn: wget failed with exit code 1 and signal 0. wget: unrecognized option: method=GET

So, further googling shows that I have busybox wget, which is not correct one for this task.
I have Alpine 3.19 in docker.

The solution is to install Gnu wget. And following commands helps:
apk update; \ apk add ca-certificates wget; \ update-ca-certificate; \

After this I cleaned nvchad and reinstalled with success.

[rdataforge]

installing curl or wget did not solve the issue. Any other suggestion?

[extde]

Here is config, which works for me (need both wget and curl, otherwise getting errors in mason). Mason here is part of NvChad.
https://github.com/extde/rust-dev-environment-docker/blob/main/Dockerfile

[josephcrawfordSRH]

LazyVim 10 also gives the same error, i am using GITHUB_TOKEN in my environment for authentication so I cannot login with gh cli.

GitHubRegistrySource(repo=mason-org/mason-registry) failed to install: Failed to fetch latest registry version from GitHub API.

How can I fix this?

[mehalter]

This is because Mason is currently not compatible with the latest neovim nightly

[josephcrawfordSRH]

oh well dang.. i didn't even think of that being an issue