SymPy core upgrade to SymEngine

One of the goals of SymEngine is to provide a backend for SymPy. This page describes a detailed plan how to make that happen.

Challenges

• SymPy core is reasonably isolated, but sometimes it calls into Python or the rest of sympy. See https://github.com/rwst/sympy-coredep/blob/master/outside.txt
• SymPy core uses caching, which essentially makes it non-deterministic.
• The C++ SymEngine has a little bit different design, where the class hierarchy does not have many methods, and all symbolic algorithms are implemented as non-member functions, which allows to decouple and isolate the symbolic algorithms. The Python wrappers should then follow this design, and not necessarily be compatible with SymPy.

Plan

1. Determine what SymPy classes and files belong to the core.

2. Create an old_core_api.py module, which will define the API to the core, the implementation will just import things from the current core, like this:

from sympy.core import Add, Mul, Integer, ...

All client code (that is, the rest of SymPy that uses the core) will access things from the core through old_core_api.py only, e.g.:

from sympy.core import Add

will be changed to

from sympy.old_core_api import Add

After this is done, the whole SymPy accesses the core through one file old_core_api.py.

3. We port symbols from old_core_api.py one by one using the following approach. We take a symbol, say Add, and create a wrapper Add that has an identical interface (the same methods and arguments), that is, each method accepts SymPy objects, converts to SymEngine, calls SymEngine's Add, and converts the result back to SymPy. Then it validates the result by calling SymPy's Add directly and compares the final expressions --- they must be identical, otherwise it will raise an exception.

   After this is done, the whole SymPy test suite passes, the actual work is done in SymEngine and in addition each result is validated with SymPy's core.

4. Remove the validation and remove the SymPy's core, that is not used at this point. Tests must still pass, since we didn't change any results from the previous step.

5. Remove the conversions SymPy -> SymEngine -> SymPy. Simply accept SymEngine.py objects and return SymEngine.py objects. Tests should still pass, since the conversion doesn't change any results.

6. At this point, SymPy is using SymEngine through SymEngine.py, but it is calling it through an old API wrapper in old_core_api.py, which is just a wrapper on top of SymEngine at this point. We can then go to SymPy, take any class or function and call SymEngine directly, because old_core_api.py is just a wrapper, but we can just as well call SymEngine directly, the result is the same.

   After this step is done, the whole of SymPy is ported on top of SymEngine.

Notes

• Each step is atomic, can be tested and make sure things are working. We can port one symbol at a time. And the process ensures that all tests will pass and nothing gets broken by the upgrade.

• Multiple people can help by simply sending PRs for one symbol at a time, or perhaps just one method of a class, say Add.diff(), at a time.

• The validation in step 3. is crucial. It ensures that our (inevitably) imperfect test suite can still be used to check that the old API implemented using SymEngine always returns the same results as SymPy core. This also checks that the caching in SymPy core works just like SymEngine, that is not cached.

• We might want to have a pure Python reference implementation of the core, that way that could be the default core in SymPy, and people can then optionally switch to use SymEngine. We need to decide whether the reference implementation should use the old API, or rather the SymEngine's API. If the old API, then we need to keep the wrapper in old_core_api.py around.