Hard-coded package and option count on search.nixos.org

[riotbib]

The message at the header saying search.nixos.org enables searching of "more than 80 000 packages" is hard-coded. Should be way higher by now, as repology also states.

[Aleksanaa]

Search more than 100,000 packages! 👍

[Sigmanificient]

Here is a small script that retrieve the number of packages from the latest nixpkgs release
It fetches the packages.json from the last item of nixpkgs s3 bucket

#!/usr/bin/env nix-shell
#!nix-shell -i bash -p curl jq brotli yq

while [ "$latest" != "null" ]; do
  prev="${latest}"
  latest=$(curl "https://nix-releases.s3.amazonaws.com/?delimiter=/&prefix=nixpkgs/&marker=${latest}" \
    | xq -rMc ".ListBucketResult.CommonPrefixes[-1].Prefix")
  echo "-> $latest"
done

target="${prev%/}"
echo "Selecting latest: $target"
count=$(curl https://releases.nixos.org/${target}/packages.json.br \
  | brotli -d --stdout | jq ".packages | length")

echo "=> $count"

For nixpkgs/nixpkgs-24.05pre606477.e976fa8f49c3 there are 111246 packages available!

For options, it is almost the same script!

--- count-pkgs.sh	2024-04-04 08:04:14.290391078 +0200
+++ count-options.sh	2024-04-04 08:10:17.967651425 +0200
@@ -3,14 +3,14 @@
 
 while [ "$latest" != "null" ]; do
   prev="${latest}"
-  latest=$(curl "https://nix-releases.s3.amazonaws.com/?delimiter=/&prefix=nixpkgs/&marker=${latest}" \
+  latest=$(curl "https://nix-releases.s3.amazonaws.com/?delimiter=/&prefix=nixos/unstable/&marker=${latest}" \
     | xq -rMc ".ListBucketResult.CommonPrefixes[-1].Prefix")
   echo "-> $latest"
 done
 
 target="${prev%/}"
 echo "Selecting latest: $target"
-count=$(curl https://releases.nixos.org/${target}/packages.json.br \
-  | brotli -d --stdout | jq ".packages | length")
+count=$(curl https://releases.nixos.org/${target}/options.json.br \
+  | brotli -d --stdout | jq "length")
 
 echo "=> $count"

Thus, for nixos/unstable/nixos-24.05pre606533.08b9151ed403 there are 18396 options available!

[riotbib]

Hej @Sigmanificient, nice work! Danke!

You may want to contact the Documentation Team, in case you want to contribute to the code base of search.nixos.org. I'll highlight @fricklerhandwerk and @infinisil of the Docu Team, in case they missed this issue for now.

[Sigmanificient]

Hej @Sigmanificient, nice work! Danke!

You may want to contact the Documentation Team, in case you want to contribute to the code base of search.nixos.org. I'll highlight @fricklerhandwerk and @infinisil of the Docu Team, in case they missed this issue for now.

I would love to contribute to the website, but unfortunately I don't know anything about elm, and I am unsure on what to exactly do.

I shared the script in case they could help open the discussion, and I would be more than happy if they turned out useful

[infinisil]

Nah the docs team doesn't have write access to search.nixos.org, that's for the marketing team :)

[riotbib]

Ah, the infamous NixOS labyrinth, thanks for the insight @infinisil :)

[Sigmanificient]

Small update: I found that https://channels.nixos.org/nixos-unstable that already redirects to the latest nixpkgs version, which means I can cut off half of the previous script!

Here is a newer version, that makes way fewer requests, and outputs both packages count and options count in a JSON format

#!/usr/bin/env nix-shell
#!nix-shell -i bash -p curl jq brotli

function parse_unstable_json() {
    local url="https://channels.nixos.org/nixos-unstable/$1.json.br"
    echo $(curl -L "$url" | brotli -d --stdout | jq "$2")
}

pkg_count=$(parse_unstable_json "packages" ".packages | length")
opt_count=$(parse_unstable_json "options" "length")

echo "{ 'packages': $pkg_count, 'options': $opt_count }" | tr "'" '"'

{ "packages": 111413, "options": 18524 }

[Sigmanificient]

This issue should be renamed Hard-coded package and option count on search.nixos.org