Requesting implementing "Integrate" method

[amcostant]

To whom it may concern,
I was wondering if there were any plans to develop an Integrate method in SymEngine. I understand that there are some workarounds for polynomial integration, but unfortunately, there is no shortcut for non-posynomials. Currently, I am having to convert to SymPy and back which is very slow, but overall faster than performing everything in SymPy.

[certik]

Thanks for the feedback @amcostant. We have done some work on the Rubi integration, see, e.g., #1497. I would like to see it finished and be able to do integrations in SymEngine. I don't have time to drive the main effort, but if anybody is interested to take a shot at that, I would be happy to help.

Update: See the comment below for the exact step by step plan how to help.

[Cazadorro]

@certik I didn't see Integrate talked about in that issue, where would I even begin to look to where I can help?

[certik]

@Cazadorro thanks for the interest. There are two avenues. One is to do what SymPy does, and implement the algorithms into SymEngine. You can start by looking at SymPy's integrate() and then try to implement some of the algorithms into SymEngine.

The other avenue is to work on the Rubi integration (which SymPy doesn't have either yet), as that will provide good integration support.

[Cazadorro]

@certik I've noticed a bunch of bugs in complicated integration with the current integration scheme in Sympy (it fails with sqrts and abs on many functions), I would prefer not to go that route if rubi integration is where everything is heading anyway. How would I help on that front? I've followed a bunch of threads and haven't really found where the "list of things we still need to do" is for rubi integration, which it looks like you are involved in some what.

[certik]

@Cazadorro I need to get up to speed on that also. @Upabjojr, do you think you could please write (or point me to if you've already done this) a simple write up with a list of steps what needs to be done to get Rubi integration into SymEngine?

[Upabjojr]

First you have to test whether the current code generator works eighth SymPy. That's probably pretty easy.

After that, you need to translate the utility functions into SymEngine.

After this is done, try to point the code generator to target C++. I expect some bugs to arise as it's not thoroughly tested.

Anyway, it's definitely faster than implementing the algorithm from scratch.

Hopefully we'll get some students for GSoC 2020.

[certik]

@Upabjojr thanks! To clarify: by "current code generator", are you talking about this file:

https://github.com/symengine/symengine/blob/ed7479baf59a36636061acc35b676bcf9073a932/symengine/utilities/matchpycpp/cpp_code_generation.py

It seems that already generates C++.

Can you point us to the SymPy utilities functions that need to be translated to SymEngine?

[Upabjojr]

Rubi has utility functions. The current python translations are in a subfolder of rubi in SymPy.

[certik]

Ok, is this the code generator that we need to get working again with SymPy: https://github.com/sympy/sympy/blob/d7a854fb789591114d7d51c3f23bba2fd4d520cc/sympy/integrals/rubi/parsetools/generate_rules.py

[certik]

Here are the utility functions that need to be translated to SymEngine: https://github.com/sympy/sympy/blob/d7a854fb789591114d7d51c3f23bba2fd4d520cc/sympy/integrals/rubi/utility_function.py

[certik]

Ok, so to start, let's start with the simplest rule file, which is this one:

https://github.com/sympy/sympy/blob/d7a854fb789591114d7d51c3f23bba2fd4d520cc/sympy/integrals/rubi/rules/piecewise_linear.py

And see if we can port it to SymEngine. It seems that one only requires a few utility functions. So that will get us started. Then we'll go from there.

[Upabjojr]

Ok, is this the code generator that we need to get working again with SymPy: https://github.com/sympy/sympy/blob/d7a854fb789591114d7d51c3f23bba2fd4d520cc/sympy/integrals/rubi/parsetools/generate_rules.py

Well, that's the code translator. It translates the rules from Mathematica to Python. After that step, the rules haven't been loaded into MatchPy yet.

You need to load them into MatchPy and then call the code generator for the decision tree (which is not the rule generator).

[certik]

I see. the piecewise_linear.py file contains the rules that go into MatchPy. So we don't need to translate that, we will use it as is. But where is the "code generator for the decision tree"? I can't find it.

[Upabjojr]

It's inside MatchPy. Some edits are necessary to use the code generator. There's an open PR somewhere.

For C++, the code generator is in symengine.

[Upabjojr]

The main problem is the usage of lambdas. When you generate the decision tree you can't find the definitions, IIRC.

[Cazadorro]

@Upabjojr Is that on SymPy or on SymEngine? Either way I still can't seem to find the PR, I found this #1482 and this #1481, neither of those mention the lambda issue.

[Upabjojr]

On SymPy.

By the way, the rules will probably need support for C++20 if ported to C++.

[certik]

@Upabjojr is this the code generator for the decision tree in matchpy: https://github.com/HPAC/matchpy/blob/a90cc4684c85f6afea0a6386d2338374eda7e1d9/matchpy/matching/code_generation.py, and here is how to use it?

[Upabjojr]

Yes, it takes a ManyToOneMatcher object as input and generates the decision tree.

It's not thoroughly tested, I fixed a bug earlier this year.

[certik]

Ok, thanks. Let's summarize the exact steps to do.

0. Currently SymPy defines ManyToOneReplacer from the Rubi rules (see here). It needs to be changed to ManyToOneMatcher, as that is what the decision tree generation in step 1. expects as input.

1. First you have to test whether the current decision tree code generator works against SymPy. That's probably pretty easy. An initial implementation already exists at [WIP] Progress of code generation sympy/sympy#15081.

2. After that, you need to translate the utility functions into SymEngine.

3. After this is done, try to point the code generator to target C++. I expect some bugs to arise as it's not thoroughly tested. The Python generator from MatchPy (in step 1.) is already ported here in SymEngine, so it needs to be tested and improved.

@Upabjojr, is this your understanding? Are the links I provided correct?

Assuming this is correct, then I would suggest to take some simple subset of Rubi rules from SymPy, say the piecewise_linear.py that I mentioned above (or even something simpler at first), then do the step 1. with it, and verify that it works. Then port only the utility functions required in piecewise_linear.py in step 2. Then do the step 3. with piecewise_linear.py and get it working in SymEngine. If that works, we can then iteratively try more and more Rubi rules and port more and more utility functions to expand the power of the integrator.

[Upabjojr]

The links are correct.

Currently SymPy defines a ManyToOneReplacer object, the replacement rules are lambdas. They need to be transformed into proper functions. There are a few issues and the generated code needs to be reviewed.

[Upabjojr]

The code generator works with ManyToOneMatcher, not with the replacer.

[certik]

Thanks, I updated the above comment with step 0.

[certik]

I updated with a link for initial implementation of step 1.

[Upabjojr]

For your information:

https://github.com/Upabjojr/rubi_generated