Precompilation fails when unit testing

[iniyannatarajan]

I am using CondaPkg to install python casa dependencies (casadata, casatools, casatasks) in my Julia package. While running unit tests from Pkg REPL (by typing test), I find that any small change in source code triggers precompilation which causes the tests to fail:

Precompiling project...
  ✗ Anime
  0 dependencies successfully precompiled in 32 seconds. 207 already precompiled.
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package
     Testing Running tests...
ERROR: LoadError: Python: ModuleNotFoundError: No module named '_image'
Python stacktrace:
 [1] import_module
   @ importlib /tmp/jl_7cFALu/.CondaPkg/env/lib/python3.8/importlib/__init__.py:127
 [2] swig_import_helper
   @ /tmp/jl_7cFALu/.CondaPkg/env/lib/python3.8/site-packages/casatools/__casac__/image.py:16
 [3] <module>
   @ /tmp/jl_7cFALu/.CondaPkg/env/lib/python3.8/site-packages/casatools/__casac__/image.py:17
 [4] <module>
   @ /tmp/jl_7cFALu/.CondaPkg/env/lib/python3.8/site-packages/casatools/image.py:4
 [5] <module>
   @ /tmp/jl_7cFALu/.CondaPkg/env/lib/python3.8/site-packages/casatools/__init__.py:43
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/1f5yE/src/err.jl:10 [inlined]
 [3] pyimport(m::String)
   @ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/concrete/import.jl:11
 [4] pyimport(::Pair{String, String})
   @ PythonCall ~/.julia/packages/PythonCall/1f5yE/src/concrete/import.jl:12
 [5] top-level scope
   @ ~/software/dev/Anime.jl/src/Anime.jl:20
 [6] include
   @ ./Base.jl:419 [inlined]
 [7] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
   @ Base ./loading.jl:1554
 [8] top-level scope
   @ stdin:1
in expression starting at /home/iniyan/software/dev/Anime.jl/src/Anime.jl:4
in expression starting at stdin:1
ERROR: LoadError: Failed to precompile Anime [628eb91b-9d1c-424a-81d5-cd7fa5d7294a] to /home/iniyan/.julia/compiled/v1.8/Anime/jl_2J6nmj.
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IO, internal_stdout::IO, keep_loaded_modules::Bool)
    @ Base ./loading.jl:1707
  [3] compilecache
    @ ./loading.jl:1651 [inlined]
  [4] _require(pkg::Base.PkgId)
    @ Base ./loading.jl:1337
  [5] _require_prelocked(uuidkey::Base.PkgId)
    @ Base ./loading.jl:1200
  [6] macro expansion
    @ ./loading.jl:1180 [inlined]
  [7] macro expansion
    @ ./lock.jl:223 [inlined]
  [8] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:1144
  [9] include(fname::String)
    @ Base.MainInclude ./client.jl:476
 [10] top-level scope
    @ none:6
in expression starting at /home/iniyan/software/dev/Anime.jl/test/runtests.jl:3
ERROR: Package Anime errored during testing

But running precompile manually in Pkg REPL mode first

(Anime) pkg> precompile
Precompiling project...
 1 dependency successfully precompiled in 19 seconds. 207 already precompiled.

and then running test succeeds, although it triggers a complete installation of the Conda environment:

     Testing Running tests...
    CondaPkg Found dependencies: /home/iniyan/software/dev/Anime.jl/CondaPkg.toml
    CondaPkg Found dependencies: /home/iniyan/.julia/packages/PythonCall/1f5yE/CondaPkg.toml
    CondaPkg Resolving changes
             + casadata (pip)
             + casatasks (pip)
             + casatools (pip)
             + libstdcxx-ng
             + pip
             + python
    CondaPkg Creating environment
             │ /home/iniyan/.julia/artifacts/a7fe6379eb776b1ec5b4ee1ddb8da76446d3f735/bin/micromamba
             │ -r /home/iniyan/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root
             │ create
             │ -y
             │ -p /tmp/jl_ePvR38/.CondaPkg/env
             │ --override-channels
             │ --no-channel-priority
             │ libstdcxx-ng[version='>=3.4,<13.0',channel='conda-forge']
             │ pip[version='>=22.0.0']
             │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*']
             │ python[version='=3.8']
             └ -c conda-forge

                                           __
          __  ______ ___  ____ _____ ___  / /_  ____ _
         / / / / __ `__ \/ __ `/ __ `__ \/ __ \/ __ `/
        / /_/ / / / / / / /_/ / / / / / / /_/ / /_/ /
       / .___/_/ /_/ /_/\__,_/_/ /_/ /_/_.___/\__,_/
      /_/

conda-forge/linux-64                                        Using cache
conda-forge/noarch                                          Using cache

Transaction

  Prefix: /tmp/jl_ePvR38/.CondaPkg/env

  Updating specs:

   - conda-forge::libstdcxx-ng[version='>=3.4,<13.0']
   - pip[version='>=22.0.0']
   - conda-forge::python[version='>=3.7,<4',build=*cpython*]
   - python===3.8


  Package               Version  Build               Channel                    Size
──────────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex           0.1  conda_forge         conda-forge/linux-64     Cached
  + _openmp_mutex           4.5  2_gnu               conda-forge/linux-64     Cached
  + bzip2                 1.0.8  h7f98852_4          conda-forge/linux-64     Cached
  + ca-certificates   2023.7.22  hbcca054_0          conda-forge/linux-64     Cached
  + ld_impl_linux-64       2.40  h41732ed_0          conda-forge/linux-64     Cached
  + libffi                3.4.2  h7f98852_5          conda-forge/linux-64     Cached
  + libgcc-ng            13.1.0  he5830b7_0          conda-forge/linux-64     Cached
  + libgomp              13.1.0  he5830b7_0          conda-forge/linux-64     Cached
  + libnsl                2.0.0  h7f98852_0          conda-forge/linux-64     Cached
  + libsqlite            3.42.0  h2797004_0          conda-forge/linux-64     Cached
  + libstdcxx-ng         12.3.0  h0f45ef3_0          conda-forge/linux-64     Cached
  + libuuid              2.38.1  h0b41bf4_0          conda-forge/linux-64     Cached
  + libzlib              1.2.13  hd590300_5          conda-forge/linux-64     Cached
  + ncurses                 6.4  hcb278e6_0          conda-forge/linux-64     Cached
  + openssl               3.1.1  hd590300_1          conda-forge/linux-64     Cached
  + pip                  23.2.1  pyhd8ed1ab_0        conda-forge/noarch       Cached
  + python               3.8.17  he550d4f_0_cpython  conda-forge/linux-64     Cached
  + readline                8.2  h8228510_1          conda-forge/linux-64     Cached
  + setuptools           68.0.0  pyhd8ed1ab_0        conda-forge/noarch       Cached
  + tk                   8.6.12  h27826a3_0          conda-forge/linux-64     Cached
  + wheel                0.41.0  pyhd8ed1ab_0        conda-forge/noarch       Cached
  + xz                    5.2.6  h166bdaf_0          conda-forge/linux-64     Cached

  Summary:

  Install: 22 packages

  Total download: 0 B

──────────────────────────────────────────────────────────────────────────────────────



Transaction starting
Linking libstdcxx-ng-12.3.0-h0f45ef3_0
Linking _libgcc_mutex-0.1-conda_forge
Linking ld_impl_linux-64-2.40-h41732ed_0
Linking ca-certificates-2023.7.22-hbcca054_0
Linking libgomp-13.1.0-he5830b7_0
Linking _openmp_mutex-4.5-2_gnu
Linking libgcc-ng-13.1.0-he5830b7_0
Linking openssl-3.1.1-hd590300_1
Linking libzlib-1.2.13-hd590300_5
Linking libffi-3.4.2-h7f98852_5
Linking bzip2-1.0.8-h7f98852_4
Linking ncurses-6.4-hcb278e6_0
Linking libuuid-2.38.1-h0b41bf4_0
Linking xz-5.2.6-h166bdaf_0
Linking libnsl-2.0.0-h7f98852_0
Linking libsqlite-3.42.0-h2797004_0
Linking tk-8.6.12-h27826a3_0
Linking readline-8.2-h8228510_1
Linking python-3.8.17-he550d4f_0_cpython
Linking wheel-0.41.0-pyhd8ed1ab_0
Linking setuptools-68.0.0-pyhd8ed1ab_0
Linking pip-23.2.1-pyhd8ed1ab_0
Transaction finished
    CondaPkg Installing Pip packages
             │ /tmp/jl_ePvR38/.CondaPkg/env/bin/pip
             │ install
             │ casadata
             │ casatasks
             └ casatools
Collecting casadata
  Obtaining dependency information for casadata from https://files.pythonhosted.org/packages/b4/9d/edf0022473b1aee4f4e9bccd5d05d24734fbce03b8de6432f1f0267124f6/casadata-2023.7.3-py3-none-any.whl.metadata
  Using cached casadata-2023.7.3-py3-none-any.whl.metadata (429 bytes)
Collecting casatasks
  Obtaining dependency information for casatasks from https://files.pythonhosted.org/packages/46/78/0770b7da2567fe99e6cb058be39fe2289d6dbe703d805bc7cfcfef66b39a/casatasks-6.5.6.22-py3-none-any.whl.metadata
  Using cached casatasks-6.5.6.22-py3-none-any.whl.metadata (666 bytes)
Collecting casatools
  Obtaining dependency information for casatools from https://files.pythonhosted.org/packages/01/e0/e54a4b9a14807c372cd85df8f1c3b6ef79ca6896424846c9a2c5320fdbc5/casatools-6.5.6.22-cp38-cp38-manylinux_2_28_x86_64.whl.metadata
  Using cached casatools-6.5.6.22-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (22 kB)
Collecting matplotlib (from casatasks)
  Obtaining dependency information for matplotlib from https://files.pythonhosted.org/packages/b4/c2/f74e0deb26379aead0956a6ecf9acd4587debba0c7abe4bd8fe53fe04ec2/matplotlib-3.7.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata
  Using cached matplotlib-3.7.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (5.6 kB)
Collecting scipy (from casatasks)
  Using cached scipy-1.10.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
Collecting certifi (from casatasks)
  Obtaining dependency information for certifi from https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl.metadata
  Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting numpy (from casatools)
  Obtaining dependency information for numpy from https://files.pythonhosted.org/packages/98/5d/5738903efe0ecb73e51eb44feafba32bdba2081263d40c5043568ff60faf/numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Collecting contourpy>=1.0.1 (from matplotlib->casatasks)
  Obtaining dependency information for contourpy>=1.0.1 from https://files.pythonhosted.org/packages/32/c8/aa9e87941002150b1a8e7087e48da1c76290268b9fdfa3034a98a5806198/contourpy-1.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached contourpy-1.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.7 kB)
Collecting cycler>=0.10 (from matplotlib->casatasks)
  Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0 (from matplotlib->casatasks)
  Obtaining dependency information for fonttools>=4.22.0 from https://files.pythonhosted.org/packages/05/68/5d6b2f770247b506e572ff83a4373a861d91eb77f722a054d2b2f2078925/fonttools-4.41.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached fonttools-4.41.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (150 kB)
Collecting kiwisolver>=1.0.1 (from matplotlib->casatasks)
  Using cached kiwisolver-1.4.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.2 MB)
Collecting packaging>=20.0 (from matplotlib->casatasks)
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting pillow>=6.2.0 (from matplotlib->casatasks)
  Obtaining dependency information for pillow>=6.2.0 from https://files.pythonhosted.org/packages/ff/8c/5927a58c43ebc16e508eef325fdc6473b569e2474d3b4be49798aa371007/Pillow-10.0.0-cp38-cp38-manylinux_2_28_x86_64.whl.metadata
  Using cached Pillow-10.0.0-cp38-cp38-manylinux_2_28_x86_64.whl.metadata (9.5 kB)
Collecting pyparsing<3.1,>=2.3.1 (from matplotlib->casatasks)
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting python-dateutil>=2.7 (from matplotlib->casatasks)
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting importlib-resources>=3.2.0 (from matplotlib->casatasks)
  Obtaining dependency information for importlib-resources>=3.2.0 from https://files.pythonhosted.org/packages/29/d1/bed03eca30aa05aaf6e0873de091f9385c48705c4a607c2dfe3edbe543e8/importlib_resources-6.0.0-py3-none-any.whl.metadata
  Using cached importlib_resources-6.0.0-py3-none-any.whl.metadata (4.2 kB)
Collecting zipp>=3.1.0 (from importlib-resources>=3.2.0->matplotlib->casatasks)
  Obtaining dependency information for zipp>=3.1.0 from https://files.pythonhosted.org/packages/8c/08/d3006317aefe25ea79d3b76c9650afabaf6d63d1c8443b236e7405447503/zipp-3.16.2-py3-none-any.whl.metadata
  Using cached zipp-3.16.2-py3-none-any.whl.metadata (3.7 kB)
Collecting six>=1.5 (from python-dateutil>=2.7->matplotlib->casatasks)
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Using cached casadata-2023.7.3-py3-none-any.whl (349.2 MB)
Using cached casatasks-6.5.6.22-py3-none-any.whl (1.8 MB)
Using cached casatools-6.5.6.22-cp38-cp38-manylinux_2_28_x86_64.whl (175.6 MB)
Using cached certifi-2023.7.22-py3-none-any.whl (158 kB)
Using cached matplotlib-3.7.2-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (9.2 MB)
Using cached numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Using cached contourpy-1.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)
Using cached fonttools-4.41.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
Using cached importlib_resources-6.0.0-py3-none-any.whl (31 kB)
Using cached Pillow-10.0.0-cp38-cp38-manylinux_2_28_x86_64.whl (3.4 MB)
Using cached zipp-3.16.2-py3-none-any.whl (7.2 kB)
Installing collected packages: casadata, zipp, six, pyparsing, pillow, packaging, numpy, kiwisolver, fonttools, cycler, certifi, scipy, python-dateutil, importlib-resources, contourpy, casatools, matplotlib, casatasks
Successfully installed casadata-2023.7.3 casatasks-6.5.6.22 casatools-6.5.6.22 certifi-2023.7.22 contourpy-1.1.0 cycler-0.11.0 fonttools-4.41.1 importlib-resources-6.0.0 kiwisolver-1.4.4 matplotlib-3.7.2 numpy-1.24.4 packaging-23.1 pillow-10.0.0 pyparsing-3.0.9 python-dateutil-2.8.2 scipy-1.10.1 six-1.16.0 zipp-3.16.2
[ Info: Load data for processing 🙆
Test Summary:    | Pass  Total   Time
Load Observation |    1      1  14.8s
     Testing Anime tests passed 

I am using CondaPkg v0.2.18 and here is my CondaPkg.toml:

[deps]
python = "=3.8"

[pip.deps]
casadata = ""
casatools = ""
casatasks = ""

Any idea what could be causing this? Am I missing something here?

[iniyannatarajan]

Some progress on this. I had to restrict casatools to a previous version in CondaPkg.toml for the precompilation to succeed. But any test case that uses casatools methods fails when I run test from Pkg REPL. I do not have any issues running these functions for production code that uses the same functions. The error always seems to be some "unknown function" causing a segfault, regardless of what method from casatools I use. Here is an example:

[12163] signal (11.1): Segmentation fault
in expression starting at /home/iniyan/software/dev/Anime.jl/test/runtests.jl:22
PyObject_GetAttr at /usr/local/src/conda/python-3.8.17/Objects/object.c:932
PyObject_GetAttr at /home/iniyan/.julia/packages/PythonCall/1f5yE/src/cpython/pointers.jl:299 [inlined]
macro expansion at /home/iniyan/.julia/packages/PythonCall/1f5yE/src/Py.jl:131 [inlined]
pygetattr at /home/iniyan/.julia/packages/PythonCall/1f5yE/src/abstract/object.jl:60
getproperty at /home/iniyan/.julia/packages/PythonCall/1f5yE/src/Py.jl:261
unknown function (ip: 0x7f3b5d96e1d6)
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
parallacticangle at /home/iniyan/software/dev/Anime.jl/src/util.jl:12
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_apply at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/builtins.c:730
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_call at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:126
eval_value at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:226
eval_body at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:478
eval_body at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:533
eval_body at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:533
jl_interpret_toplevel_thunk at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:762
jl_toplevel_eval_flex at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:912
jl_toplevel_eval_flex at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
include_string at ./loading.jl:1903
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
_include at ./loading.jl:1963
include at ./client.jl:478
unknown function (ip: 0x7f3b6d0fd062)
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
do_call at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:126
eval_value at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:226
eval_stmt_value at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:177 [inlined]
eval_body at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:624
jl_interpret_toplevel_thunk at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/interpreter.c:762
jl_toplevel_eval_flex at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:912
jl_toplevel_eval_flex at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:856
ijl_toplevel_eval_in at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/toplevel.c:971
eval at ./boot.jl:370 [inlined]
exec_options at ./client.jl:280
_start at ./client.jl:522
jfptr__start_43375.clone_1 at /home/iniyan/software/installables/julia-1.9.2/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2758 [inlined]
ijl_apply_generic at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/gf.c:2940
jl_apply at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/julia.h:1879 [inlined]
true_main at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/jlapi.c:573
jl_repl_entrypoint at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/src/jlapi.c:717
main at /cache/build/default-amdci5-2/julialang/julia-release-1-dot-9/cli/loader_exe.c:59
unknown function (ip: 0x7f3b84c29d8f)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x401098)
Allocations: 43417916 (Pool: 43392383; Big: 25533); GC: 75
ERROR: Package Anime errored during testing (received signal: 11)