MitM use case

[florian-rabe]

@nthiery @markuspf @kohlhase @tkw1536 @Jazzpirate

I've talked to everybody over the last few days to coordinate how we setup our MitM use case, both in terms of short-term demoability and long-term scalability.

The best design appears to be as follows:

• SageMath, GAP, and MMT run on a dedicated server (the MitM server).
• SageMath and GAP maintain SCSCP connections to MMT, over which MMT sends requests and receives responses.
• MMT connects to LMFDb as needed via the LMFDb HTTP API.
• MMT exposes an SCSCP interface to the outside, over which user systems (e.g., a SageMath running on a user's computer) can send requests to the MitM server. MMT splits these requests into computation tasks as necessary, which are forwarded to other systems on the MitM server (e.g., GAP or SageMath) or external systems (e.g., LMFDb).

To make this work, (at least) the following tasks remain for us:

1. SageMath:

   • @nthiery produces a docker file for Sage that includes the appropriate branch of py-openmath as well of py-scscp.
   • Due to a weakness in the SCSCP specification, py-scscp currently expects a whitelist based on (cdbase,cd,name) triples (see line 62 of demo-server.py).
     @nthiery should change that to a whitelist based on cdbase alone so that he can whitelist all SageMath symbols at once.
   • @nthiery reports the results (in particular the docker file and how to start the server) to @tkw1536.

2. Gap

   • @markuspf produces a docket file for GAP.
     -[ ] This file should include an SCSCP-server answering at the TCP interface that Markus has exposed already.
   • @tkw1536 has already modified MMT to allow for JSON-based SCSCP. Therefore, @markuspf only has to provide relatively minor control operations (initialization etc.) @markuspf and @tkw1536 should look at this together. If necessary, @tkw1536 can modify MMT's SCSCP interface with GAP-specific tricks to make it easier for @markuspf .
   • @markuspf reports the results (in particular the docker file and how to start the server) to @tkw1536.

3. MitM server setup

   • @tkw1536 set up the MitM server using the above docker files as well as MMT.
   • @tkw1536 sets up auto-build features that automatically update the server if the respective docker files change. That way @nthiery and @markuspf do not need to access of modify the MitM server.
   • @tkw1536 documents the high-level interface of the MitM server for developers. This must include instructions how to test use cases and log the individual objects exchanged between components.
   • @tkw1536 reports the results to @florian-rabe and @Jazzpirate.

4. Use case

   • @nthiery, @markuspf, @Jazzpirate document the SageMath code for starting a use case as well as the intermediate objects that are passed around. @Jazzpirate collects these.
   • @Jazzpirate starts with the use cases and sets up the necessary alignments.
   • @florian-rabe coordinates with @tkw1536, @nthiery, @markuspf, @Jazzpirate about any bugs or problems that come up.

[nthiery]

Thanks Florian for the detailed battle plan; that's very helpful.

On Fri, Jun 29, 2018 at 07:37:13AM -0700, Florian Rabe wrote: + ***@***.*** produces a docker file for Sage that includes the appropriate branch of py-openmath as well of py-scscp. + Due to a weakness in the SCSCP specification, py-scscp currently expects a whitelist based on (cdbase,cd,name) triples (see line 62 of demo-server.py). ***@***.*** should change that to a whitelist based on cdbase alone so that he can whitelist all SageMath symbols at once.

I have been working on this yesterday: I actually just changed the demo server by adding two additional heads: - eval: takes an OpenMath expression, evaluate it to a Sage object and convert back to OpenMath (which may trigger arbitrary calculations) - apply: takes a list of OpenMath expressions, and apply the first one (it should evaluate to a Python function) with the remaining objects as arguments. Does this cover the needs? Cheers, Nicolas -- Nicolas M. Thiéry "Isil" <nthiery@users.sf.net> http://Nicolas.Thiery.name/

[kohlhase]

@florian-rabe thanks for the detailed plan. There is one piece missing: in the MACIS use case, we also used Singular. And that needs to be integrated as well. @markuspf, could you please report on this?
If I remember correctly there was a simple python wrapper for Singular that could probably be updated to a SCSCP wrappper.

[tkw1536]

@nthiery that sounds like exactly what we need. Can you commit that code and include it in your Dockerfile?

[nthiery]

Hi @tkw1536,
The code is now committed. See the following demo on binder:

https://mybinder.org/v2/gh/OpenDreamKit/MitM-Sage/master?filepath=sage/demo_sage_scscp.ipynb

The notebook manually launches a background Sage SCSCP server, and connects to it to run some calculations.

To setup a docker image running just the server, you should be able to reuse the git repo as is:

   https://github.com/OpenDreamKit/MitM-Sage/

up to uncommenting the CMD line at the end of the Dockerfile to automatically launch the server (it will run on port 26133, customizable in demo_sage_scscp_server.py.

I haven't tested it though; feel free to ping back if you encounter problem. Presumably, we should have a separate git repo with Dockerfile with the exact minimum to build the docker image. Let me know how you want to proceed with that.

[tkw1536]

@nthiery That sounds like great work.

I would want to auto-build the docker image on every commit, and automatically update the container on our server. Which means I would want to commit the Dockerfile with a server start script and appropriate EXPOSE directive.

Can you give me access so that we can commit that to a branch of the repository?

Furthermore, I do not believe that Binder auto-builds an image on every commit and also pushes that to a docker registry (or does it?), so I would suggest using DockerHub Automated Builds for this. With repository access, I should be able to set up a post-commit hook.

[nthiery]

Can you give me access so that we can commit that to a branch of the repository?

Done!

Furthermore, I do not believe that Binder auto-builds an image on every commit

No indeed; it only rebuilds an image on the fly if someone requests it. And the image is only pushed to its local registry. DockerHub should be fine indeed; the build time is reasonable. Of course we will still need to merge master into the branch on a regular basis. Btw: this repo originally was on git@gl.mathhub.info:ODK/Sage.git; I have been using github for binder purposes lately and haven't updated the original repo for a while. But I could if this can make things any easier.

[kohlhase]

Btw: this repo originally was on git@gl.mathhub.info:ODK/Sage.git; I have been using github for binder purposes lately and haven't updated the original repo for a while. But I could if this can make things any easier.

For now, (in the experiment phase) this is fine. But I am wondering what the right way to do this will be. Intuitively,

• the Sage code extensions should become part of Sage, so that they do not decay and are maintained together with the system and
• the generated CDs should be on MathHub/ODK/Sage so that they become part of the processing pipeline.

I wonder whether that would be possible to set up?

[markuspf]

I think it'd be better if I run the GAP server on field for the development phase, because that provides faster turnaround times (for me) for debugging.

It's probably not a problem to provide a Docker thingy once the code is stable enough.

[nthiery]

For now, (in the experiment phase) this is fine. But I am wondering what the right way to do this will be. Intuitively, * the Sage code extensions should become part of Sage, so that they do not decay and are maintained together with the system and * the generated CDs should be on MathHub/ODK/Sage so that they become part of the processing pipeline. I wonder whether that would be possible to set up?

Ok. Note that a good deal of the Sage code extensions is actually generic for Python, and implemented in py-openmath and py-scscp. Whose long term maintenance we thus need to consider as well.

[kohlhase]

you are right, I was aware of this.