Julia/CondaPkg.jl doesn't install a Julia package required by Python dependency #95

dingraha · 2023-06-12T17:45:13Z

Big fan of PythonCall.jl, JuliaCall.jl, etc.. Thanks so much for these packages.

In my application, I have a chain of tools like this:

OpenMDAOCore.jl is a pure-Julia package, with no dependencies other than Julia itself.
omjlcomps is a Python package that depends on OpenMDAOCore.jl. I use juliapkg to specify that dependency via a juliapkg.json file: https://github.com/byuflowlab/OpenMDAO.jl/blob/master/python/omjlcomps/juliapkg.json
OpenMDAO.jl is a Julia package that depends on both OpenMDAOCore.jl and omjlcomps. The CondaPkg.toml file declares omjlcomps as a dependency: https://github.com/byuflowlab/OpenMDAO.jl/blob/master/julia/OpenMDAO.jl/CondaPkg.toml

The problem: if I create a clean Julia environment and try to install OpenMDAO.jl, I get an error when I attempt to import OpenMDAO.jl. For some reason omjlcomps can't find OpenMDAOCore.jl. Here's what that looks like:

conda-forge/linux-64                                32.1MB @   2.9MB/s 11.5s

Transaction

  Prefix: /tmp/jl_qAPRx5/.CondaPkg/env

  Updating specs:

   - conda-forge::libstdcxx-ng[version='>=3.4,<13.0']
   - openmdao[version='>=3.26.0,<4']
   - pip[version='>=22.0.0']
   - conda-forge::python[version='>=3.7,<4',build=*cpython*]


  Package                Version  Build                Channel                    Size
────────────────────────────────────────────────────────────────────────────────────────
  Install:
────────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex            0.1  conda_forge          conda-forge/linux-64     Cached
  + _openmp_mutex            4.5  2_gnu                conda-forge/linux-64     Cached
  + brotli                 1.0.9  h166bdaf_8           conda-forge/linux-64     Cached
  + brotli-bin             1.0.9  h166bdaf_8           conda-forge/linux-64     Cached
  + bzip2                  1.0.8  h7f98852_4           conda-forge/linux-64     Cached
  + ca-certificates     2023.5.7  hbcca054_0           conda-forge/linux-64     Cached
  + certifi             2023.5.7  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + charset-normalizer     3.1.0  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + idna                     3.4  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + ld_impl_linux-64        2.40  h41732ed_0           conda-forge/linux-64     Cached
  + libblas                3.9.0  17_linux64_openblas  conda-forge/linux-64     Cached
  + libbrotlicommon        1.0.9  h166bdaf_8           conda-forge/linux-64     Cached
  + libbrotlidec           1.0.9  h166bdaf_8           conda-forge/linux-64     Cached
  + libbrotlienc           1.0.9  h166bdaf_8           conda-forge/linux-64     Cached
  + libcblas               3.9.0  17_linux64_openblas  conda-forge/linux-64     Cached
  + libexpat               2.5.0  hcb278e6_1           conda-forge/linux-64     Cached
  + libffi                 3.4.2  h7f98852_5           conda-forge/linux-64     Cached
  + libgcc-ng             13.1.0  he5830b7_0           conda-forge/linux-64     Cached
  + libgfortran-ng        13.1.0  h69a702a_0           conda-forge/linux-64     Cached
  + libgfortran5          13.1.0  h15d22d2_0           conda-forge/linux-64     Cached
  + libgomp               13.1.0  he5830b7_0           conda-forge/linux-64     Cached
  + liblapack              3.9.0  17_linux64_openblas  conda-forge/linux-64     Cached
  + libnsl                 2.0.0  h7f98852_0           conda-forge/linux-64     Cached
  + libopenblas           0.3.23  pthreads_h80387f5_0  conda-forge/linux-64     Cached
  + libsqlite             3.42.0  h2797004_0           conda-forge/linux-64     Cached
  + libstdcxx-ng          12.3.0  h0f45ef3_0           conda-forge/linux-64     Cached
  + libuuid               2.38.1  h0b41bf4_0           conda-forge/linux-64     Cached
  + libzlib               1.2.13  hd590300_5           conda-forge/linux-64     Cached
  + ncurses                  6.4  hcb278e6_0           conda-forge/linux-64     Cached
  + networkx                 3.1  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + numpy                 1.24.3  py311h64a7726_0      conda-forge/linux-64     Cached
  + openmdao              3.26.0  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + openssl                3.1.1  hd590300_1           conda-forge/linux-64     Cached
  + packaging               23.1  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + pip                   23.1.2  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + platformdirs           3.5.3  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + pooch                  1.7.0  pyha770c72_3         conda-forge/noarch       Cached
  + pysocks                1.7.1  pyha2e5f31_6         conda-forge/noarch       Cached
  + python                3.11.4  hab00c5b_0_cpython   conda-forge/linux-64     Cached
  + python_abi              3.11  3_cp311              conda-forge/linux-64     Cached
  + readline                 8.2  h8228510_1           conda-forge/linux-64     Cached
  + requests              2.31.0  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + scipy                 1.10.1  py311h64a7726_3      conda-forge/linux-64     Cached
  + setuptools            67.7.2  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + tk                    8.6.12  h27826a3_0           conda-forge/linux-64     Cached
  + typing-extensions      4.6.3  hd8ed1ab_0           conda-forge/noarch       Cached
  + typing_extensions      4.6.3  pyha770c72_0         conda-forge/noarch       Cached
  + tzdata                 2023c  h71feb2d_0           conda-forge/noarch       Cached
  + urllib3                2.0.3  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + wheel                 0.40.0  pyhd8ed1ab_0         conda-forge/noarch       Cached
  + xz                     5.2.6  h166bdaf_0           conda-forge/linux-64     Cached

  Summary:

  Install: 51 packages

  Total download: 0 B

────────────────────────────────────────────────────────────────────────────────────────



Transaction starting
Linking libstdcxx-ng-12.3.0-h0f45ef3_0
Linking _libgcc_mutex-0.1-conda_forge
Linking libgfortran5-13.1.0-h15d22d2_0
Linking python_abi-3.11-3_cp311
Linking ld_impl_linux-64-2.40-h41732ed_0
Linking ca-certificates-2023.5.7-hbcca054_0
Linking libgomp-13.1.0-he5830b7_0
Linking libgfortran-ng-13.1.0-h69a702a_0
Linking _openmp_mutex-4.5-2_gnu
Linking libgcc-ng-13.1.0-he5830b7_0
Linking libbrotlicommon-1.0.9-h166bdaf_8
Linking libopenblas-0.3.23-pthreads_h80387f5_0
Linking openssl-3.1.1-hd590300_1
Linking libzlib-1.2.13-hd590300_5
Linking libffi-3.4.2-h7f98852_5
Linking bzip2-1.0.8-h7f98852_4
Linking ncurses-6.4-hcb278e6_0
Linking libuuid-2.38.1-h0b41bf4_0
Linking libexpat-2.5.0-hcb278e6_1
Linking xz-5.2.6-h166bdaf_0
Linking libnsl-2.0.0-h7f98852_0
Linking libbrotlienc-1.0.9-h166bdaf_8
Linking libbrotlidec-1.0.9-h166bdaf_8
Linking libblas-3.9.0-17_linux64_openblas
Linking libsqlite-3.42.0-h2797004_0
Linking tk-8.6.12-h27826a3_0
Linking readline-8.2-h8228510_1
Linking brotli-bin-1.0.9-h166bdaf_8
Linking libcblas-3.9.0-17_linux64_openblas
Linking liblapack-3.9.0-17_linux64_openblas
Linking brotli-1.0.9-h166bdaf_8
Linking tzdata-2023c-h71feb2d_0
Linking python-3.11.4-hab00c5b_0_cpython
Linking wheel-0.40.0-pyhd8ed1ab_0
Linking setuptools-67.7.2-pyhd8ed1ab_0
Linking pip-23.1.2-pyhd8ed1ab_0
Linking typing_extensions-4.6.3-pyha770c72_0
Linking charset-normalizer-3.1.0-pyhd8ed1ab_0
Linking pysocks-1.7.1-pyha2e5f31_6
Linking idna-3.4-pyhd8ed1ab_0
Linking certifi-2023.5.7-pyhd8ed1ab_0
Linking packaging-23.1-pyhd8ed1ab_0
Linking networkx-3.1-pyhd8ed1ab_0
Linking typing-extensions-4.6.3-hd8ed1ab_0
Linking urllib3-2.0.3-pyhd8ed1ab_0
Linking platformdirs-3.5.3-pyhd8ed1ab_0
Linking requests-2.31.0-pyhd8ed1ab_0
Linking pooch-1.7.0-pyha770c72_3
Linking numpy-1.24.3-py311h64a7726_0
Linking scipy-1.10.1-py311h64a7726_3
Linking openmdao-3.26.0-pyhd8ed1ab_0
Transaction finished
    CondaPkg Installing Pip packages
             │ /tmp/jl_qAPRx5/.CondaPkg/env/bin/pip
             │ install
             │ juliapkg ~=0.1.10
             └ omjlcomps ~=0.2.0
Collecting juliapkg~=0.1.10
  Using cached juliapkg-0.1.10-py3-none-any.whl (15 kB)
Collecting omjlcomps~=0.2.0
  Using cached omjlcomps-0.2.3-py3-none-any.whl (13 kB)
Collecting semantic-version~=2.9 (from juliapkg~=0.1.10)
  Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: openmdao~=3.26.0 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from omjlcomps~=0.2.0) (3.26.0)
Collecting juliacall~=0.9.13 (from omjlcomps~=0.2.0)
  Using cached juliacall-0.9.13-py3-none-any.whl (11 kB)
Requirement already satisfied: networkx>=2.0 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (3.1)
Requirement already satisfied: numpy in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (1.24.3)
Requirement already satisfied: scipy in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (1.10.1)
Requirement already satisfied: requests in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (2.31.0)
Requirement already satisfied: packaging in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from openmdao~=3.26.0->omjlcomps~=0.2.0) (23.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from requests->openmdao~=3.26.0->omjlcomps~=0.2.0) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from requests->openmdao~=3.26.0->omjlcomps~=0.2.0) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from requests->openmdao~=3.26.0->omjlcomps~=0.2.0) (2.0.3)
Requirement already satisfied: certifi>=2017.4.17 in /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages (from requests->openmdao~=3.26.0->omjlcomps~=0.2.0) (2023.5.7)
Installing collected packages: semantic-version, juliapkg, juliacall, omjlcomps
Successfully installed juliacall-0.9.13 juliapkg-0.1.10 omjlcomps-0.2.3 semantic-version-2.10.0
ERROR: InitError: Python: Julia: ArgumentError: Package OpenMDAOCore not found in current path.
- Run `import Pkg; Pkg.add("OpenMDAOCore")` to install the OpenMDAOCore package.
Stacktrace:
  [1] macro expansion
    @ ./loading.jl:1163 [inlined]
  [2] macro expansion
    @ ./lock.jl:223 [inlined]
  [3] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [4] eval
    @ ./boot.jl:368 [inlined]
  [5] eval
    @ ./Base.jl:65 [inlined]
  [6] pyjlmodule_seval(self::Module, expr::PythonCall.Py)
    @ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/jlwrap/module.jl:13
  [7] _pyjl_callmethod(f::Any, self_::Ptr{PythonCall.C.PyObject}, args_::Ptr{PythonCall.C.PyObject}, nargs::Int64)
    @ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/jlwrap/base.jl:62
  [8] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
    @ PythonCall.C ~/.julia/packages/PythonCall/1f5yE/src/cpython/jlwrap.jl:47
  [9] PyImport_Import
    @ ~/.julia/packages/PythonCall/1f5yE/src/cpython/pointers.jl:299 [inlined]
 [10] pyimport(m::String)
    @ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/concrete/import.jl:11
 [11] __init__()
    @ OpenMDAO ~/projects/pythoncall_openmdao/dev/OpenMDAO.jl-clean/julia/OpenMDAO.jl/src/OpenMDAO.jl:14
 [12] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base ./loading.jl:831
 [13] _tryrequire_from_serialized(pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:978
 [14] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1347
 [15] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
 [16] macro expansion
    @ ./loading.jl:1180 [inlined]
 [17] macro expansion
    @ ./lock.jl:223 [inlined]
 [18] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
 [19] eval
    @ ./boot.jl:368 [inlined]
 [20] eval_user_input(ast::Any, backend::REPL.REPLBackend)
    @ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:151
 [21] repl_backend_loop(backend::REPL.REPLBackend)
    @ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:247
 [22] start_repl_backend(backend::REPL.REPLBackend, consumer::Any)
    @ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:232
 [23] run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool)
    @ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:369
 [24] run_repl(repl::REPL.AbstractREPL, consumer::Any)
    @ REPL ~/local/julia/1.8.5/share/julia/stdlib/v1.8/REPL/src/REPL.jl:355
 [25] (::Base.var"#967#969"{Bool, Bool, Bool})(REPL::Module)
    @ Base ./client.jl:419
 [26] #invokelatest#2
    @ ./essentials.jl:729 [inlined]
 [27] invokelatest
    @ ./essentials.jl:726 [inlined]
 [28] run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool)
    @ Base ./client.jl:404
 [29] exec_options(opts::Base.JLOptions)
    @ Base ./client.jl:318
 [30] _start()
    @ Base ./client.jl:522
Python stacktrace:
 [1] seval
   @ ~/.julia/packages/PythonCall/1f5yE/src/jlwrap/module.jl:25
 [2] <module>
   @ /tmp/jl_qAPRx5/.CondaPkg/env/lib/python3.11/site-packages/omjlcomps/__init__.py:9
Stacktrace:
  [1] pythrow()
    @ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/err.jl:94
  [2] errcheck
    @ ~/.julia/packages/PythonCall/1f5yE/src/err.jl:10 [inlined]
  [3] pyimport(m::String)
    @ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/concrete/import.jl:11
  [4] __init__()
    @ OpenMDAO ~/projects/pythoncall_openmdao/dev/OpenMDAO.jl-clean/julia/OpenMDAO.jl/src/OpenMDAO.jl:14
  [5] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
    @ Base ./loading.jl:831
  [6] _tryrequire_from_serialized(pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:978
  [7] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1347
  [8] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [9] macro expansion
    @ ./loading.jl:1180 [inlined]
 [10] macro expansion
    @ ./lock.jl:223 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
during initialization of module OpenMDAO

julia>

You also might be able to see the error from the AutoMerge workflow for the Julia General registry: https://github.com/JuliaRegistries/General/actions/runs/5244999369/jobs/9471800986?pr=85376.

I was able to work around this by just explicitly adding OpenMDAOCore.jl using Pkg in OpenMDAO.jl's __init__():
https://github.com/byuflowlab/OpenMDAO.jl/blob/892afe54b78dca25a6b66237ed24a584f5dc1c47/julia/OpenMDAO.jl/src/OpenMDAO.jl#L15
But I wonder if I'm doing something wrong, or if there's a better way?

The text was updated successfully, but these errors were encountered:

cjdoris · 2023-06-15T08:09:56Z

Yeah CondaPkg/JuliaPkg doesn't support recursive Julia -> Python -> Julia dependencies. It's too hard.

I think the best solution right now is to add OpenMDAOCore as a dependency to OpenMDAO.

dingraha · 2023-06-15T10:27:28Z

Gotcha, thanks. To be clear, OpenMDAOCore.jl was always a dependency of OpenMDAO.jl. The error occurs when installing OpenMDAO.jl in an environment where OpenMDAOCore.jl isn't explicitly installed (i.e., where OpenMDAOCore.jl isn't in the environment's Project.toml). The solution I came up with was adding this to OpenMDAO.jl:

function __init__()
    PythonCall.pycopy!(om, PythonCall.pyimport("openmdao.api"))
    Pkg.add("OpenMDAOCore")
    PythonCall.pycopy!(omjlcomps, PythonCall.pyimport("omjlcomps"))
end

which I think installs OpenMDAOCore.jl in the current environment just before attempting to import the python package omjlcomps.

I'm fine with that solution, just want to make sure that's what you're suggesting.

cjdoris · 2023-06-16T15:03:00Z

Oh I see! Even though OpenMDAOCore is a dep of OpenMDAO, if you install OpenMDAO into some other project, then import OpenMDAOCore fails because it's not a direct dependency in the project. That is annoying. I'm not sure there's anything I could change in PythonCall to make this work, but will have a think.

You could use Base.require to import OpenMDAOCore instead. This circumvents these checks.

Edit: I'd recommend against stuff like calling Pkg from __init__.

cjdoris · 2023-06-16T15:06:27Z

Also, is it important to have these 3 separate packages. Could you instead have one OpenMDAO package with all of this stuff in?

dingraha · 2023-06-16T20:01:17Z

You could use Base.require to import OpenMDAOCore instead. This circumvents these checks.

Interesting, I haven't heard of Base.require. How would I do that? I see it takes a module and a symbol (the name of the module I want to load, :OpenMDAOCore in this case?) https://docs.julialang.org/en/v1/base/base/#Base.require. What would the first argument be, and where would I put it? At the top of OpenMDAO.jl?

Also, is it important to have these 3 separate packages. Could you instead have one OpenMDAO package with all of this stuff in?

That's how it was set up originally, but it wasn't great for a couple of reasons. First, I think putting it all in one package would lead to a circular dependency:

The Python package omjlcomps needs access to the methods defined in OpenMDAOCore.jl, so it's a dependency.
The Julia package OpenMDAO.jl needs the classes defined in the Python package omjlcomps.

So if all the Julia code was in just one package, omjlcomps would need OpenMDAO.jl, and OpenMDAO.jl would need omjlcomps. That actually seemed to work, though I would get scary warnings re: the OpenMDAO.jl module being loaded twice.

Also, with the way things are right now, OpenMDAOCore.jl is a very lightweight dependency—actually has no dependencies other than Julia itself. So it allows Julia package developers to support OpenMDAO without messing around with the omjlcomps Python package, or Python at all, which is nice.

cjdoris · 2023-06-22T16:28:57Z

All valid reasons.

I think you can do something like

Base.require(Base.PkgId("name", "uuid"))

to import a specific package.

dingraha · 2023-06-27T19:34:34Z

Gotcha, I tried

Base.require(Base.PkgId(Base.UUID("24d19c10-6eee-420f-95df-4537264b2753"), "OpenMDAOCore"))

at both the top level of the "parent" package OpenMDAO and inside OpenMDAO's __init__(), but still get the same error.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Julia/CondaPkg.jl doesn't install a Julia package required by Python dependency #95

Julia/CondaPkg.jl doesn't install a Julia package required by Python dependency #95

dingraha commented Jun 12, 2023

cjdoris commented Jun 15, 2023

dingraha commented Jun 15, 2023

cjdoris commented Jun 16, 2023 •

edited

cjdoris commented Jun 16, 2023

dingraha commented Jun 16, 2023

cjdoris commented Jun 22, 2023

dingraha commented Jun 27, 2023

Julia/CondaPkg.jl doesn't install a Julia package required by Python dependency #95

Julia/CondaPkg.jl doesn't install a Julia package required by Python dependency #95

Comments

dingraha commented Jun 12, 2023

cjdoris commented Jun 15, 2023

dingraha commented Jun 15, 2023

cjdoris commented Jun 16, 2023 • edited

cjdoris commented Jun 16, 2023

dingraha commented Jun 16, 2023

cjdoris commented Jun 22, 2023

dingraha commented Jun 27, 2023

cjdoris commented Jun 16, 2023 •

edited