Incorrect cuda-nvcc version

[ymtoo]

To install cuda-toolkit 11.7.1:

julia> CondaPkg.add("cuda-toolkit"; version="11.7.1", channel = "nvidia/label/cuda-11.7.1")

(test-condapkg) pkg> conda st
CondaPkg Status /home/ymtoo/Projects/tmp/test-condapkg/CondaPkg.toml
Environment
  /home/ymtoo/Projects/tmp/test-condapkg/.CondaPkg/env
Packages
  cuda-toolkit v11.7.1 (11.7.1, channel=nvidia/label/cuda-11.7.1)

shell> .CondaPkg/env/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:12:58_PDT_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0

The installed packages:

$ conda list
# packages in environment at /home/ymtoo/Projects/tmp/test-condapkg/.CondaPkg/env:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_sysroot_linux-64_curr_repodata_hack 3                   h69a702a_13    conda-forge
alsa-lib                  1.2.8                h166bdaf_0    conda-forge
binutils                  2.40                 hdd6e379_0    conda-forge
binutils_impl_linux-64    2.40                 hf600244_0    conda-forge
binutils_linux-64         2.40                 hbdbef99_1    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-compiler                1.6.0                hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cuda-cccl                 12.0.90              ha770c72_1    conda-forge
cuda-cccl-impl            2.0.1                ha770c72_0    conda-forge
cuda-cccl_linux-64        12.0.90              ha770c72_1    conda-forge
cuda-command-line-tools   11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-compiler             12.0.0               h9bef630_0    conda-forge
cuda-cudart               12.0.107             h59595ed_6    conda-forge
cuda-cudart-dev           12.0.107             h59595ed_6    conda-forge
cuda-cudart-dev_linux-64  12.0.107             h59595ed_6    conda-forge
cuda-cudart-static        12.0.107             h59595ed_6    conda-forge
cuda-cudart-static_linux-64 12.0.107             h59595ed_6    conda-forge
cuda-cudart_linux-64      12.0.107             h59595ed_6    conda-forge
cuda-cuobjdump            12.0.76              h59595ed_0    conda-forge
cuda-cupti                12.0.90              h59595ed_1    conda-forge
cuda-cuxxfilt             12.0.76              h59595ed_0    conda-forge
cuda-documentation        11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-driver-dev           12.0.107             h59595ed_6    conda-forge
cuda-driver-dev_linux-64  12.0.107             h59595ed_6    conda-forge
cuda-gdb                  12.0.90              hd47b8d6_0    conda-forge
cuda-libraries            12.0.0               ha770c72_1    conda-forge
cuda-libraries-dev        12.0.0               ha770c72_1    conda-forge
cuda-memcheck             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nsight               12.0.78              ha770c72_0    conda-forge
cuda-nsight-compute       11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-nvcc                 12.0.76             hba56722_10    conda-forge
cuda-nvcc-dev_linux-64    12.0.76              ha770c72_0    conda-forge
cuda-nvcc-impl            12.0.76              h59595ed_0    conda-forge
cuda-nvcc-tools           12.0.76              h59595ed_0    conda-forge
cuda-nvcc_linux-64        12.0.76             h8a487aa_10    conda-forge
cuda-nvdisasm             12.0.76              h59595ed_0    conda-forge
cuda-nvml-dev             12.0.76              hcb278e6_0    conda-forge
cuda-nvprof               12.0.90              h59595ed_0    conda-forge
cuda-nvprune              12.0.76              h59595ed_0    conda-forge
cuda-nvrtc                12.0.76              h59595ed_1    conda-forge
cuda-nvrtc-dev            12.0.76              h59595ed_1    conda-forge
cuda-nvtx                 12.0.76              hcb278e6_0    conda-forge
cuda-nvvp                 12.0.90              h59595ed_0    conda-forge
cuda-opencl               12.0.76              h59595ed_0    conda-forge
cuda-opencl-dev           12.0.76              ha770c72_0    conda-forge
cuda-profiler-api         12.0.76              ha770c72_0    conda-forge
cuda-sanitizer-api        12.0.90              h59595ed_0    conda-forge
cuda-toolkit              11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-tools                11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-version              12.0                 hffde075_2    conda-forge
cuda-visual-tools         11.7.1                        0    nvidia/label/cuda-11.7.1
cxx-compiler              1.6.0                h00ab1b0_0    conda-forge
dbus                      1.13.6               h5008d03_3    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
gcc                       12.3.0               h8d2909c_1    conda-forge
gcc_impl_linux-64         12.3.0               he2b93b0_0    conda-forge
gcc_linux-64              12.3.0               h76fc315_1    conda-forge
gds-tools                 1.5.0.59             hcb278e6_0    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gxx                       12.3.0               h8d2909c_1    conda-forge
gxx_impl_linux-64         12.3.0               he2b93b0_0    conda-forge
gxx_linux-64              12.3.0               h8a814eb_1    conda-forge
icu                       72.1                 hcb278e6_0    conda-forge
kernel-headers_linux-64   3.10.0              h4a8ded7_13    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.19.3               h08a2579_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libcublas                 12.0.1.189           hcb278e6_2    conda-forge
libcublas-dev             12.0.1.189           hcb278e6_2    conda-forge
libcufft                  11.0.0.21            hcb278e6_1    conda-forge
libcufft-dev              11.0.0.21            hcb278e6_1    conda-forge
libcufile                 1.5.0.59             hcb278e6_0    conda-forge
libcufile-dev             1.5.0.59             hcb278e6_0    conda-forge
libcurand                 10.3.1.50            hcb278e6_0    conda-forge
libcurand-dev             10.3.1.50            hcb278e6_0    conda-forge
libcusolver               11.4.2.57            hcb278e6_1    conda-forge
libcusolver-dev           11.4.2.57            hcb278e6_1    conda-forge
libcusparse               12.0.0.76            hcb278e6_1    conda-forge
libcusparse-dev           12.0.0.76            hcb278e6_1    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-devel_linux-64     12.3.0               h8bca6fd_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libglib                   2.76.4               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libnpp                    12.0.0.30            h59595ed_0    conda-forge
libnpp-dev                12.0.0.30            h59595ed_0    conda-forge
libnuma                   2.0.16               h0b41bf4_1    conda-forge
libnvjitlink              12.0.76              hcb278e6_1    conda-forge
libnvjitlink-dev          12.0.76              hcb278e6_1    conda-forge
libnvjpeg                 12.0.0.28            hcb278e6_0    conda-forge
libnvjpeg-dev             12.0.0.28            ha770c72_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsanitizer              12.3.0               h0f45ef3_0    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libstdcxx-devel_linux-64  12.3.0               h8bca6fd_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxkbcommon              1.5.0                h5d7e998_3    conda-forge
libxkbfile                1.1.0                h166bdaf_1    conda-forge
libxml2                   2.11.4               h0d562d8_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nsight-compute            2022.4.0.15          h1166182_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.89                 he45b914_0    conda-forge
ocl-icd                   2.3.1                h7f98852_0    conda-forge
openssl                   3.1.2                hd590300_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
sysroot_linux-64          2.17                h4a8ded7_13    conda-forge
wayland                   1.22.0               h8c25dac_1    conda-forge
xcb-util                  0.4.0                hd590300_1    conda-forge
xcb-util-image            0.4.0                h8ee46fc_1    conda-forge
xcb-util-keysyms          0.4.0                h8ee46fc_1    conda-forge
xcb-util-renderutil       0.3.9                hd590300_1    conda-forge
xcb-util-wm               0.4.1                h8ee46fc_1    conda-forge
xkeyboard-config          2.39                 hd590300_0    conda-forge
xorg-compositeproto       0.4.2             h7f98852_1001    conda-forge
xorg-damageproto          1.2.1             h7f98852_1002    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxcomposite        0.4.6                h0b41bf4_1    conda-forge
xorg-libxdamage           1.1.5                h7f98852_1    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrandr            1.5.2                h7f98852_1    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-libxtst              1.2.3             h7f98852_1002    conda-forge
xorg-randrproto           1.5.0             h7f98852_1001    conda-forge
xorg-recordproto          1.14.2            h7f98852_1002    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-util-macros          1.19.3               h7f98852_0    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge

By adding channels = ["anaconda"] to the CondaPkg.toml, the correct cuda-nvcc version is installed.

julia> conda up

(test-condapkg) pkg> conda st
CondaPkg Status /home/ymtoo/Projects/tmp/test-condapkg/CondaPkg.toml
Environment
  /home/ymtoo/Projects/tmp/test-condapkg/.CondaPkg/env
Packages
  cuda-toolkit v11.7.1 (11.7.1, channel=nvidia/label/cuda-11.7.1)
Channels
  anaconda

shell> .CondaPkg/env/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

The installed packages:

$ conda list
# packages in environment at /home/ymtoo/Projects/tmp/test-condapkg/.CondaPkg/env:
#
# Name                    Version                   Build  Channel
cuda-cccl                 11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-command-line-tools   11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-compiler             11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-cudart               11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-cudart-dev           11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-cuobjdump            11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-cupti                11.7.101                      0    nvidia/label/cuda-11.7.1
cuda-cuxxfilt             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-documentation        11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-driver-dev           11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-gdb                  11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-libraries            11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-libraries-dev        11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-memcheck             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nsight               11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nsight-compute       11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-nvcc                 11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-nvdisasm             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nvml-dev             11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nvprof               11.7.101                      0    nvidia/label/cuda-11.7.1
cuda-nvprune              11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nvrtc                11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-nvrtc-dev            11.7.99                       0    nvidia/label/cuda-11.7.1
cuda-nvtx                 11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-nvvp                 11.7.101                      0    nvidia/label/cuda-11.7.1
cuda-sanitizer-api        11.7.91                       0    nvidia/label/cuda-11.7.1
cuda-toolkit              11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-tools                11.7.1                        0    nvidia/label/cuda-11.7.1
cuda-visual-tools         11.7.1                        0    nvidia/label/cuda-11.7.1
gds-tools                 1.3.1.18                      0    nvidia/label/cuda-11.7.1
libcublas                 11.10.3.66                    0    nvidia/label/cuda-11.7.1
libcublas-dev             11.10.3.66                    0    nvidia/label/cuda-11.7.1
libcufft                  10.7.2.91                     0    nvidia/label/cuda-11.7.1
libcufft-dev              10.7.2.91                     0    nvidia/label/cuda-11.7.1
libcufile                 1.3.1.18                      0    nvidia/label/cuda-11.7.1
libcufile-dev             1.3.1.18                      0    nvidia/label/cuda-11.7.1
libcurand                 10.2.10.91                    0    nvidia/label/cuda-11.7.1
libcurand-dev             10.2.10.91                    0    nvidia/label/cuda-11.7.1
libcusolver               11.4.0.1                      0    nvidia/label/cuda-11.7.1
libcusolver-dev           11.4.0.1                      0    nvidia/label/cuda-11.7.1
libcusparse               11.7.4.91                     0    nvidia/label/cuda-11.7.1
libcusparse-dev           11.7.4.91                     0    nvidia/label/cuda-11.7.1
libnpp                    11.7.4.75                     0    nvidia/label/cuda-11.7.1
libnpp-dev                11.7.4.75                     0    nvidia/label/cuda-11.7.1
libnvjpeg                 11.8.0.2                      0    nvidia/label/cuda-11.7.1
libnvjpeg-dev             11.8.0.2                      0    nvidia/label/cuda-11.7.1
nsight-compute            2022.2.1.3                    0    nvidia/label/cuda-11.7.1

Julia and the package version:

julia> versioninfo()
Julia Version 1.9.2
Commit e4ee485e909 (2023-07-05 09:39 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 8 × Intel(R) Core(TM) i7-10510U CPU @ 1.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 8 virtual cores

(test-condapkg) pkg> st
Status `~/Projects/tmp/test-condapkg/Project.toml`
  [992eb4ea] CondaPkg v0.2.18

(test-condapkg) pkg> st --manifest
Status `~/Projects/tmp/test-condapkg/Manifest.toml`
  [992eb4ea] CondaPkg v0.2.18
  [692b3bcd] JLLWrappers v1.4.1
  [0f8b85d8] JSON3 v1.13.2
  [0b3b1443] MicroMamba v0.1.14
  [69de0a69] Parsers v2.7.2
  [fa939f87] Pidfile v1.3.0
  [aea7be01] PrecompileTools v1.1.2
  [21216c6a] Preferences v1.4.0
  [6c6a2e73] Scratch v1.2.0
  [856f2bd8] StructTypes v1.10.0
  [f8abcde7] micromamba_jll v1.4.9+0
  [0dad84c5] ArgTools v1.1.1
  [56f22d72] Artifacts
  [2a0f44e3] Base64
  [ade2ca70] Dates
  [f43a241f] Downloads v1.6.0
  [7b1f6079] FileWatching
  [b77e0a4c] InteractiveUtils
  [4af54fe1] LazyArtifacts
  [b27032c2] LibCURL v0.6.3
  [76f85450] LibGit2
  [8f399da3] Libdl
  [56ddb016] Logging
  [d6f4376e] Markdown
  [a63ad114] Mmap
  [ca575930] NetworkOptions v1.2.0
  [44cfe95a] Pkg v1.9.2
  [de0858da] Printf
  [3fa0cd96] REPL
  [9a3f8284] Random
  [ea8e919c] SHA v0.7.0
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [fa267f1f] TOML v1.0.3
  [a4e569a6] Tar v1.10.0
  [8dfed614] Test
  [cf7118a7] UUIDs
  [4ec0a83e] Unicode
  [deac9b47] LibCURL_jll v7.84.0+0
  [29816b5a] LibSSH2_jll v1.10.2+0
  [c8ffd9c3] MbedTLS_jll v2.28.2+0
  [14a3606d] MozillaCACerts_jll v2022.10.11
  [83775a58] Zlib_jll v1.2.13+0
  [8e850ede] nghttp2_jll v1.48.0+0
  [3f19e933] p7zip_jll v17.4.0+0

[cjdoris]

What about channels = ["nvidia"] or channels = ["nvidia/label/cuda-11.7.1"]?

[cjdoris]

The underlying issue is that CondaPkg passes -c conda-forge if you don't specify any channels, and so it's finding a bunch of compatible packages from the conda-forge channel with higher version numbers, and therefore installing those.

Currently CondaPkg specifies --no-channel-priority meaning that all channels are given equal priority. A future CondaPkg could specify a priority (e.g. put conda-forge at the bottom) which could mean that nvidia has higher priority than conda-forge.

[ymtoo]

channels = ["nvidia/label/cuda-11.7.1"] yields the correct cuda-nvcc version. For the following CondaPkg.toml

channels = ["nvidia/label/cuda-11.7.1", "conda-forge"]

[deps.cuda-toolkit]
channel = "nvidia/label/cuda-11.7.1"
version = "11.7.1"

conda up installs the wrong cuda-nvcc. This prevents the installation of other packages from conda-forge.

To clarify, CondaPkg installs the correct cuda-toolkit version from "nvidia/label/cuda-11.7.1" but it installs the dependencies, e.g., cuda-nvcc, with higher version numbers from "conda-forge". Am I right?