Expand testing

[MilesCranmer]

Currently the test suite is very small, most notably the juliacall suite: https://github.com/JuliaPy/PythonCall.jl/blob/main/pytest/test_all.py which is only 19 lines long. I am very worried about reliability issues from this and whether this might result in regressions.

I think expanding the test suite should be a priority, especially on the juliacall side, before new features are worked on.

Here are some suggestions. Feel free to check any that are already done or add more

• Unittests for features shown in the docs
  • ALL JuliaCall guide examples (https://juliapy.github.io/PythonCall.jl/stable/juliacall/)
  • ALL PythonCall guide examples (https://juliapy.github.io/PythonCall.jl/stable/pythoncall/)
  • ALL conversion rules Python -> Julia (https://juliapy.github.io/PythonCall.jl/stable/conversion-to-julia/)
  • ALL conversion rules Julia -> Python (https://juliapy.github.io/PythonCall.jl/stable/conversion-to-python/)
• Miscellaneous unittests
  • GC tests (currently there are zero)
  • Dummy package with pyjuliapkg and newmodule used
  • Multithreading unittests (with PythonCall.GC.disable())
    • Need to launch additional test suite in CI, with signals turned on
  • Distributed compute unittests
  • Julia's stdlib tests from Python via juliacall (copy the test suite; disable everything, and gradually turn on tests)
  • Python's stdlib tests from Julia via PythonCall.jl (copy the test suite; disable everything, and gradually turn on tests)
  • Any areas missing due to coverage milestones:
• Code coverage milestones
  • Find way to track Python code generated from Julia code
  • Reach 60% code coverage on Python source code
  • Reach 80% code coverage on Python source code
  • Reach 90% code coverage on Python source code
  • Reach 100% code coverage on Python source code
  • Set up code coverage for Julia source code (currently not tracked)
  • Reach 60% code coverage on Julia source code
  • Reach 80% code coverage on Julia source code
  • Reach 90% code coverage on Julia source code
  • Reach 100% code coverage on Julia source code
• Add integration tests
  • Numpy
  • PythonPlot.jl
  • Pandas (https://juliapy.github.io/PythonCall.jl/stable/compat/#Tabular-data-/-Pandas)
  • SymPy.jl (whenever PythonCall.jl version is made)
  • Matplotlib interactive support (harder to test, but you would want to test the features explained here – https://juliapy.github.io/PythonCall.jl/stable/compat/#MatPlotLib-/-PyPlot-/-Seaborn)
  • IPython support
  • PySR
  • diffeqpy
  • PyTorch import issue (basically just verify the solution on https://juliapy.github.io/PythonCall.jl/stable/faq/)

[cjdoris]

Absolutely, I've been holding off making another release because I recently did a big refactor and want to get test coverage up so I can trust I didn't break anything.

Indeed the test suite of JuliaCall is short - but then so is JuliaCall itself. My priority has been to get the coverage for PythonCall up, because that covers most of JuliaCall too.

Your list looks really good. I don't think it's especially needed to test the same code twice (from Julia and from Python) so most of the points in the first group should probably done in the PythonCall tests in Julia, leaving the tests in Python to the parts of JuliaCall actually written in Python.

Integration tests sound great.

[MilesCranmer]

Sounds good to me!