SCDoc: add MathJax for rendering math

[prko]

ADDED :

1. As this PR has been modified several times, along with many discussions and some reviews, please read the following post on this PR:
   SCDoc: add MathJax for rendering math #6260 (comment)

2. A summary of the main objection to the initial commit of this PR and the author's reaction:
   SCDoc: add MathJax for rendering math #6260 (comment)

3. The state as of around 17 May 2024:
   SCDoc: add MathJax for rendering math #6260 (comment)

   • This PR
     • SCDoc modal tag math:: is restored
     • MathJax package minimised and optimised
     • Due to Debian policy, MathJax is not included in Debian when building from source.
       (done by adding code to CMakeLists.txt and tested successfully)
     • To do:
       A MathJax package that does not conflict with Debian policy should be automatically installed when building.
       Currently, using npm is the easiest way, at least from the point of view of the author of this PR.
   • @capital-G's suggestion
     • MathML via temml: not so successful
     • KaTeX is being tested
   • @scztt's suggestion
     • use a cached version on Debian for Debian policy
   • @smoge
     • mentioned diagram feature of LaTex

---

INITIAL POST:

With this PR, we can represent mathematical expressions using LaTeX and MathML without an internet connection.

• MathJax supports LaTeX and MathML, so I also combined them into a PR.
• I think I have included the minimal files needed to use MathJax offline.

Purpose and Motivation

I have spent a lot of time using SC-Doc to write course material for my Digital Sound Production lecture, as SuperCollider is one of the key tools in the course. My students have suffered from the lack of a reader friendly layout in SCdoc and I have also found that there are some editing features missing in SCdoc. One of these is the lack of mathematical expressions. Currently, to include reader-friendly mathematical expressions, we have to use graphics from other software such as LaTeX, but this is time consuming and tedious to edit. With this PR, we can save a lot of time and energy, and productivity will increase.

In the following thread are some reactions to the idea of implementing LaTeX and MathML in SCdoc:
https://scsynth.org/t/scdoc-schelp-style-add-on-latex-mathml-subsubsection-multimedia-file-iframe-sub-sup-small-line-height-hr/9242

This PR is a revised PR of the relevant part of the following PR #6254
I think I have incorporated all the opinions of @nhthn and @jrsurge in this new PR.

Types of changes

• Documentation
• New feature

To-do list

• Code is tested
• All tests are passing
• Updated documentation
• This PR is ready for review

[prko]

Converted PDF from the corresponding reference in SCDoc included in this PR:
LaTeX and MathML Support via MathJax _ SuperCollider 3.14.0-dev Help.pdf

[mtmccrea]

Thanks for your work on this @prko!

In the doc renderer, *renderSubTree, is there a reason to be parsing math within most of the modal tags? It seems that's a lot more text being parsed and string-matched, etc, than necessary.

Would it make more sense to implement a new modal tag, such as math::, to then only send that enclosed text through the math parser? I'd imagine the implementation would be like the code:: modal tag, which supports both inline and block rendering.

In that case, the user would specify, e.g., math::a^2 + b^2 = c^2:: rather than \(a^2 + b^2 = c^2\) (the necessary bounding charaters would be inserted by the renderer before forwarding to the mathjax parser).

I can't comment on the how the new library is included because I'm unfamiliar with how that can/should work.

[prko]

In the doc renderer, *renderSubTree, is there a reason to parse maths in most modal tags?

In the original PR (#6254), the maths parsing was in `*renderToFile':
https://github.com/prko/supercollider/blob/288a1c80bd9ee09793c9ffa5969c17dadddb858c/SCClassLibrary/SCDoc/SCDocRenderer.sc#L1193-L1269

As I read @Nathn's review:

I have my doubts over whether SCDoc should be expanded to keep glomming on new features, but more importantly: I'm afraid modifying the generated HTML with tons of .replace calls isn't the right way to do this. The correct way to extend SCDoc syntax is to modify the SCDoc parser and HTML generator in the SCDoc directory. Unfortunately, as written, this is very fragile and hacky.

I was not sure what the following sentence meant:

The correct way to extend the SCDoc syntax is to modify the SCDoc parser and HTML generator in the SCDoc directory.

I interpreted it to mean that the parser should be inside appropriate modal tags. (Whether the maths parser should be inside \strong and \soft tags is not clear, but I think strong and soft might be needed sometimes).

By moving the maths parser to *renderSubTree, it currently skips some text in an SCDoc when parsing maths expressions, whereas the maths parser in #6254 parses maths expressions by checking all the text in an SCdoc because it is in *renderToFile. So the current implementation is economical in parsing maths expressions, but the structure of SCDocRenderer.sc is complicated, while the implementation in #6254 is economical in the structure of SCDocRenderer.sc, but complicated in parsing maths expressions.

It seems that's a lot more text being parsed and string-matched, etc, than necessary.

Yes, this is true! The parser parseMathML should only work for the MathML part. It should not try to parse the LaTex part or any other part.

Would it make more sense to implement a new modal tag, such as math::, to then only send that enclosed text through the math parser? I'd imagine the implementation would be like the code:: modal tag, which supports both inline and block rendering.

You are absolutely right, but adding a new SCDoc modal tag is hard coded, and it seems to require adding a lexer in some cpp, hpp, l and y files in the source. See: #6131. You can see what I did to add the subsubsection tag in SCDoc and I had no success as I do not understand C++.

So the current way works best from my amateur programming level.

Unless someone who can modify the appropriate parts of the source file wants to implement MathJax in SCDoc, I think a soft-coded implementation similar to the way suggested in this PR seems to be the only way to implement this feature.

In that case, the user would specify, e.g., math::a^2 + b^2 = c^2:: rather than (a^2 + b^2 = c^2) (the necessary bounding charaters would be inserted by the renderer before forwarding to the mathjax parser).

I want to say four things:

1. We do not need to do anything for LaTeX, since the LaTeX part is handled by the script in the following part:
   https://github.com/supercollider/supercollider/blob/a1f3baccf5c594c9d5e73fcf0af75d0e661aa94c/SCClassLibrary/SCDoc/SCDocRenderer.sc#L254C1-L276C19

2. In LaTeX, mathematical expressions can be presented as inline maths with text in a paragraph, and as block maths separate from a paragraph. In your proposal, I do not know how to display the inline maths and the block maths. The default is

   • inline: \(a^2 + b^2 = c^2\)
   • block:\[a^2 + b^2 = c^2\]

   As mentioned above, no extra tag is needed to use LaTeX in SCDoc, as far as I know.

3. I think that implementing math:: requires modifying the C++ source code, but implementing :math: can be achieved by modifying this PR. By using :math: and :\math:, only the string within these words can be attempted to be parsed. This reduces unnecessary attempts to parse MathML strings. One problem here is that it is difficult to do this correctly if there is more than one pair of :math: and :/math: in the text. For example, 000:math:111:\math:222:math:333:\math:444 can be recognised as a single pair: The part :math:111:\math:222:math:333:\math: can therefore be regarded as a mathematical expression. I spent almost two weeks trying to solve it before giving up. Maybe it would be easier to merge @JordanHendersonMusic's PR.

4. Adding a tag like math:: or :math: can reduce unnecessary parsing when rendering SCDoc, but users should type these tags additionally. Which is better, a user-friendly environment or a CPU-friendly environment?

Anyway. I will think about how to reduce unnecessary parsing attempts...

[nhthn]

My honest opinion is that this is a really bad idea from the get-go.

SC is already tremendously bloated in terms of dependencies. This makes the problem worse. It makes every SC download from now on bundle MathJax with it. (Also much better to make it a git submodule than check in all the files, but that's beside the point.)

SCDoc is not a professional markup language, it's a simple and lightweight system fulfilling the basic needs of SC and its users. (If you need a full-featured markup language, use any reStructuredText or Markdown parser out there.) I am completely unconvinced of the demand for MathJax as a feature and I recommend that this PR be closed. I am strongly opposed to the addition of MathJax to SCDoc at all, in any form.

[prko]

@nhthn
I followed your feedback on #6254:

Not a good idea -- this makes SCDoc connect to the Internet.

So I included MathJax.... Now you say again that including MathJax is a bad idea. This is sad to hear, but I still want to learn and find a better way.

(Also much better to make it a git submodule than check in all the files, but that's beside the point.)).

Could you let me know if there is a way to use the git submodule within SCDoc?

(If you need a full-featured markup language, use any reStructuredText or Markdown parser out there.)

It is sad for me. I have already written more than 800 pages with SCDoc...

For me, SC-Doc is not bad because I can create teaching material for online access from SuperCollider HelpBrowser and for PDF. I think I can combine these PDFs as one and use it for a book publication. The problem with SCDoc is that it lacks readability and some features. So I am trying to improve the SCDoc feature.

If you think I am making a wrong attempt, could you please tell me a good markup language to do it like SCDoc with a good feature to print as a paper book?

I do not think it is such a good idea to convert more than 800 pages into another language, but I should change if I am doing completely wrong things (for me and also for the development of SC. I thought it might be a good idea to help improve SCDoc.).

I am completely unconvinced of the demand for MathJax as a feature and I recommend that this PR be closed. I am strongly opposed to the addition of MathJax to SCDoc at all, in any form.

OK, I see what you think, but I'd like to keep this open for at least a couple of weeks to see what other people think.

[mtmccrea]

FWIW I think there are a number of people that would find math rendering support very useful.

We knew that the devil would be in the details of how MathJax could be supported offline/static, how heavy it is, and what the maintenance overhead would be. This was an important proof-of-concept to show what can be done, and I appreciate the work that @prko has put into this!

I think it's worth keeping the PR open to hear more thoughts if people have them, especially if someone can detail why this specific library can or cannot be maintained—I'm unfamiliar with this in practice and I'm not gaining any insight so far from the argument that "dependencies are bad". Is it all or nothing?

It makes every SC download from now on bundle MathJax with it.

Is this the case? It looks like only a subset is required, as reflected in @prko's implementation.

[redFrik]

maybe i'm missing something, but why not just insert the math formulas as png images? like for example in the Shaper helpfile.

[prko]

@redFrik
There are a few reasons. At least for me:

1. The use of various mathematical symbols in sclang is currently very limited.

The pi symbol can be used by copying it from the internet or a character map, but other symbols can only be used via an image, but the image cannot be placed in a paragraph with text.

2. Mathematical expressions in sclang are not so ideal for readability when inline with the flow of text in a paragraph.

3. Images cannot be inline with the text flow in a paragraph.

4. Using LaTeX via MathJax in sclang dramatically reduces the amount of work and makes the workflow more flexible.

I have already created about 750 image files for use in various SCDocs:

• About 100 Latex files and the rendered SVG files from them.
• About 650 screenshots of Dorico, Finale, MuseScore, the plot of sclang, etc.

Taking screenshots and saving them as images requires extra work:

• Saving a screenshot file in a subfolder of the existing schelp file.
• Finding the file and write the file path in schelp file.
• Rendering SCDoc.

It takes extra time and increases the number of image files. Some operations were not too bad. However, some were very time consuming and it is a big barrier to writing documents.

If mathematical expressions can be done in sclang, we can use that time for something else, because using LaTeX via MathJax in sclang dramatically reduces the amount of work and makes the workflow more flexible:

[muellmusik]

Just to echo @prko that it's great you've put so much work into this, and even if it doesn't go forward, I hope it's at least been useful in developing your skills.

I think I agree that we should be careful about adding a new dependency. It might be better as some have suggested to look at converting the doc to a different format that provides builtin support for math symbols. If you're interested in that, I think it would be best to agree a strategy before investing too much effort. This would necessarily involve a lot of automated conversion. I think we'd need to provide simple tools for Quark authors etc to convert to minimise friction.

If that could be done in a way which reduced overall maintenance demands, I think you'd have a runner.

[mtmccrea]

@smoge Thanks for considering the specifics of the proposal more closely.

The sooner we can arrive at an optimal way to include the library, the conversation can be grounded in the practical issues of maintenance. Implementation on the SCDoc renderer side is secondary.

• Including MathJax does not introduce significant dependencies. It can be included as a .js file and it operates independently without needing additional libraries or frameworks.

Could you elaborate with regard to the way this PR is currently including the library?
See the inclusion of HelpSource/lib/node_modules/mathjax-full/es5/ in this PR. Is this unnecessary?
Note we're requiring a fully offline implementation.

• Keeping the library up-to-date is straightforward. But even if it is not updated often, nothing will explode or break. Most probably nothing will happen.

I'm unfamiliar with web rendering tech. What aspects of this library might require updating in the future? Given that it's rendering to HTML, I'd imagine this doesn't evolve much over time, and we might even be able to pin to a particular version/submodule and it will "just work" into the future? (i.e. no updates needed?)

[smoge]

Choosing one or the other is moot if you get them both for the same implementation/maintenance burden.

MathJax Components

To make using MathJax easier in web pages, the various pieces that make up MathJax have been packaged into separate files called “components”. For example, there is a component for MathML input, and one for SVG output, and the various TeX extensions are packaged as separate components. You can mix and match the various components to customize MathJax to suit your particular needs (this is described in detail in the section on Configuring MathJax below); the individual component files that you specify are loaded when MathJax starts up.

There are also components that combine several others into one larger file that loads everything you need to run MathJax all at once. These represent some of the standard combinations of input and output formats, and you will probably find one of these that suits your needs. You can configure the various components in order to customize how they run, even when they are loaded as part of a combined component. For example, you can set the delimiters to be used for in-line and displayed math for the TeX input component whether the TeX component was loaded individually, or as part of the tex-chtml component.

It is even possible for you to create your own components or custom builds of MathJax, or incorporate the MathJax components into larger files that contain other assets your website might need (see the section on Making a Custom Build of MathJax for more details).

[smoge]

We can make one file just with what we will need. That will be the most efficient way.

It does not need anything more. No node.js, no internet connection, nothing, just the help browser.

@prko already did a proof-of-concept, so we know it works.

[prko]

On the MathJax version of this PR

@smoge

I quickly checked the documentation, there are two versions. Version 2 is divided into several js files. Version 3 is just one file. But, contrary to what some would think, version 3 loads faster. I believe @prko picked version 2.

The MathJax version of this PR is version 3.

According to the following link:
https://docs.mathjax.org/en/latest/web/hosting.html#getting-mathjax-via-npm

I downloaded it by running the following command in a terminal:

npm install mathjax-full@3

@smoge

We can make one file just with what we will need. That will be the most efficient way.

Basically, we may only need the following two files, which are referenced in this PR:

• /lib/polyfill.min.js
  

  supercollider/SCClassLibrary/SCDoc/SCDocRenderer.sc

  Line 360 in f084ac8

  << "<script src='" << baseDir << "/lib/polyfill.min.js?features=es6'></script>\n" // local access /* << "<script src='https://polyfill.io/v3/polyfill.min.js?features=es6'></script>\n" */ // online access

• /lib/node_modules/mathjax-full/es5/tex-mml-chtml.js
  

  supercollider/SCClassLibrary/SCDoc/SCDocRenderer.sc

  Line 380 in f084ac8

  << "<script id='MathJax-script' src='" << baseDir << "/lib/node_modules/mathjax-full/es5/tex-mml-chtml.js' async></script>\n" // offline accesss /* << "<script id='MathJax-script' src='https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js' async></script>\n" */ // online accesss

However, I am not sure if I can delete all the other files as I seem to have found some links to other folders in the file tex-mml-chtml.js.

I think we need to inspect the inside of this file, but I do not understand javascript.

Maintaining MathJax

As far as I understand, there is only one thing to maintain to implement MathJax into SuperCollider: Linking the updated version of the updated version of MathJax if we want.

The current official version of MathJax is 3, but MathJax 2 is still hosted online by the MathJax development team. They have also released version 4 beta.

MathJax will be included in the SC release if this PR is approved. Then, in theory, we won't have to do any maintenance, unless at least one user wants to upgrade the MathJax in SC from version 3 to any future version, or the use of version 3 is forbidden by the MathJax development team. As a composer and music theorist, MathJax 3 is already too rich for me. So, at least for me, I won't want to upgrade MathJax from 3 to 4, because I don't know the deep level of mathematics.

On removing MathML support

@mtmccrea

@prko
Is there a reason to support both LaTeX and MathML, considering they both achieve the same goal of rendering math?

@smoge

MathML is not really for humans to type.

I also thought it would be better to remove MathML, because it is too long to type and too complicated to use with a reference manual. The reason I did not remove it is only because of the time I spent investigating it: when I could not configure LaTeX to use teletype text, MathML worked. So I added the necessary code to make MathML work with .replace' and .replaceRegexp'. Now I have configured LaTeX to work in teletype, soft, strong and emphasis text. So theoretically I can remove MathML. However, I have not tested large expressions yet, and MathML is also a standard for mathematical expressions in HTML format, so I am still hesitant to remove MathML in this PR.

@mtmccrea

@prko
Would supporting one or the other make the footprint of this library smaller or make the implementation in the SCDoc renderer simpler?

Choosing one or the other is moot if you get them both for the same implementation/maintenance burden.

By removing MathML support, I basically need just the following lines:

supercollider/SCClassLibrary/SCDoc/SCDocRenderer.sc

Line 359 in f084ac8

// MathJax |--->

to

supercollider/SCClassLibrary/SCDoc/SCDocRenderer.sc

Line 381 in f084ac8

// <---| MathJax

Also the following code and its related parts are needed to write the documentation as SCDoc:
https://github.com/supercollider/supercollider/blob/f084ac86f22c6619bd71bcbaa78e43483817a7b3/SCClassLibrary/SCDoc/SCDocRenderer.sc#L82-86

 	*teletypeWithinCode { |str|
 		^str
 		.replace(":teletype:", "teletype::")
 		.replace(":/teletype:", "::")
 	}

On using LaTeX in SCDoc

@smoge

Just pick LaTeX, with the SCDoc markup (math:: ::). That's about it.

1. To use LaTeX in SCDoc we do not need anything else. The other things I did are for MathML and writing schelpfile.

2. To use inline math we should use \( ... \), and to use display math we should use \[ ... \]. I also read $...$ for inline math and $$...$$ for display math, but they do not work.

3. Have you tried using math:: ... ::? I cannot use it because sclang gives me the following error:

   ERROR: In C:\Temp\SuperCollider for Windows (x64)\HelpSource\Guides\WritingHelp_Punctuation_HTML_LaTeX_MathML.schelp:
    At line 1572: syntax error, unexpected ::, expecting end of file
   

   To use SCDoc markup, I think we need to modify many parts of C++. Below is the relevant part from the parts I wrote earlier in this PR:

   You are absolutely right, but adding a new SCDoc modal tag is hard coded, and it seems to require adding a lexer in some cpp, hpp, l and y files in the source. See: adding subsubsection to the table of contents in schelp documents #6131. You can see what I did to add the subsubsection tag in SCDoc and I had no success as I do not understand C++.

   I think that implementing math:: requires modifying the C++ source code, but implementing :math: can be achieved by modifying this PR. By using :math: and :\math:, only the string within these words can be attempted to be parsed. This reduces unnecessary attempts to parse MathML strings. One problem here is that it is difficult to do this correctly if there is more than one pair of :math: and :/math: in the text. For example, 000:math:111:\math:222:math:333:\math:444 can be recognised as a single pair: The part :math:111:\math:222:math:333:\math: can therefore be regarded as a mathematical expression. ...

On the need to implement MathJax in SuperCollider

@smoge

If that's the real discussion, if SuperCollider needs that or not, let's talk about the real issue then. Let's discuss how the community uses SC, or if there would be some benefits, for researchers and students.

Adding a tag like math:: or :math: can reduce unnecessary parsing when rendering SCDoc, but users should type these tags additionally. Which is better, a user-friendly environment or a CPU-friendly environment?

I would say that there are many advantages to implementing MathJax in SuperCollider. However, I would shorten the list of advantages of implementing MathMax in SC by mentioning only the first one: it is possible to write beautiful and readable help documents.

@JordanHendersonMusic

Teaching is one of the places where this is really important, mostly for accessibility, so I definitely get where this is coming from, but supercollider is already a vast and ambitious project. Supercollider might already be too vast?

As @JordanHendersonMusic mentioned, teaching SC is already a huge task, but I think it would be more enjoyable if we could have a "readability-improved" help document.

Please see the attached images below, comparing the left (using LaTeX via MathJax) and the right (current help documents) on each image. Which style would be better for SC newcomers?

[smoge]

@prko I think we agreed on all the important points. Right?

We need to discuss the implementation. I think we can clean up and simplify a lot. I've written about it above.

Let's keep just math:: :: on the SCDoc markup side, and just LaTex on the javascript side. And that will be it. Let's clean all the parsers, we don't need that.

Also, use the asynchronous configuration for the library, it will make the pages load as fast as they do today, and the math will be rendered with lower priority.

The code will be more beautiful, trust me and see.

No regexp, no replace. Just write as any other SCDoc markup, and the library will do the hard work for you. I can't see why we would need anything more complicated than that.

[smoge]

@prko Another thing. Let's try not to change too many help files. Let's just add this new feature in pages where it is important, for example, all digital filters and chaotic equations, etc.

Don't try to change every square root and exponent in every file. No need for that. Try to be more minimalistic now.

Let me know if you need anything. If you want I'm available to review as well. Good luck, comrade.

[mtmccrea]

To use LaTeX in SCDoc we do not need anything else.

I also thought it would be better to remove MathML, because it is too long to type and too complicated to use with a reference manual.

I suggest we then support only LaTeX. (We can expand in the future if there is interest.)

Now I have configured LaTeX to work in teletype, soft, strong and emphasis text. So theoretically I can remove MathML.

Even better. But anyway I think rendering LaTeX is not necessary in these styling modal tags, and removing it should lower the number of times the latex parser is called. (Again, could be added later if desired.) I suggest we keep this PR minimally viable. Relatedly:

Let's try not to change too many help files.

In fact, I'd say just do the Help file instructing on the use of the feature, and maybe one other Help file as an example. Leave other documentation changes to a future PR.

Basically, we may only need the following two files, which are referenced in this PR:
/lib/polyfill.min.js
/lib/node_modules/mathjax-full/es5/tex-mml-chtml.js

Can this be reflected in an update to this PR?

However, I am not sure if I can delete all the other files as I seem to have found some links to other folders in the file tex-mml-chtml.js.
I think we need to inspect the inside of this file, but I do not understand javascript.

If you need help, I hope some others in this conversation who understand the details can assist!

[prko]

@capital-G

Here is a summary of the main opposition before getting a response from the lead developer of MathJax:
(This is generated by an AI tool. I have tested more than 10 AI tools to do this. This is the easiest and best. However, AI only summarises the first few days ...., fortunately most of the important objections have been included).

• prko submitted a pull request to add support for rendering LaTeX and MathML mathematical expressions in SCDoc (SuperCollider's documentation format) using the MathJax library. The goal was to allow including math expressions without having to rely on external graphics.

• mtmccrea reviewed the changes and suggested implementing a new modal tag like math:: to avoid unnecessarily parsing math in most modal tags. prko explained the challenges in modifying the SCDoc parser to add new tags.

• nhthn expressed opposition to adding MathJax as a dependency to SuperCollider, feeling it added bloat and that SCDoc should remain lightweight. They recommended closing the PR.

• prko responded to nhthn, explaining he had included MathJax files to avoid requiring an internet connection as nhthn had previously suggested. He asked for recommendations on alternative markup languages if SCDoc was not suitable.

• mtmccrea commented that math rendering would be useful to many and the PR was a valuable proof-of-concept. He felt it was worth discussing further, especially regarding the specifics of including the MathJax library.

• redFrik asked why math expressions couldn't just be included as PNG images.

• prko explained the limitations of using images and how MathJax would provide a much more flexible and efficient workflow.

• muellmusik suggested considering converting the docs to a different format with built-in math support, but agreed a clear strategy was needed first. Automated conversion tools would be required.

• smoge reviewed the changes and discussed with mtmccrea and prko some of the technical considerations around efficiently including the needed parts of the MathJax library.

[capital-G]

@smoge

But, to be frank, I'm not participating anymore in any discussion because there was never a real one. I just don't have the motivation. Sorry.

I am sorry to hear this and I can relate to your frustration. I still would really like and invite you to be part of participation as your input in discussion, code and motivation is really valuable and appreciated!

I just want to find a good modus operandi together to tackle "controversial" topics such as this b/c I believe that just maintaining a status quo will just lead to a decay of contributions.

Yet we should find and settle for a way that new features don't generate problems later down the road - we should be aware of them so we can make hopefully a good decision on this.

As this was a feature before which was removed because of issues, people voiced concerns.

@prko

Thank you for this summary! Maybe edit the top post to include it so others are able to catch up with the current state?

1. The use of MathML, due to the complexity of the code syntax, was not welcomed by some users in the responses to the original commit of this PR.

I think the direct use and embedding of MathML is indeed a bit strange and would limit schelp to HTML output. But that's where transpilers such as couldtemml help which could translate tex to MathML on the website. As MathML uses the HTML5 standard to display formula, it is a lot less heavyweight than MathJax.

2. I should mention here that there was an objection to using MathJax's online script

From my understanding the objection was regarding the necessity for online access in order to view the documentation properly.
Relying on an outside tool/dependency such as an internet connection to view the documentation is something different than to edit or build it.

3. that uses GitHub's mathematical function and presentation template.

I think you refer to the MathML export?
It would be possible, but the source code would not be not in such a good state.

4. since the reason for MathJax's objection was its maintenance. After posting the response from the lead developer of MathJax, this issue no longer seems to be an issue, because by shipping the MathJax 3.2.2 library with SC, we do not need to worry about maintenance until at least one user insists on using MathJax 4.x. How about `temml'?

Although npm-world is a rather short-living environment, once something runs in the browser it gets broken or deprecated if we have a static js file. We are still using a 6 year old jquery and codemirror and it is still working and not generating (many) problems.
An additional point for consideration: If math/tex support breaks, how much else breaks? In this case: only the rendering of formulas within docs, nothing else (as we do not rely on it to build other structures). So I think it is quite manageable to maintain this and in the worst case just "delete" it and have some "tex" code floating around.

6. Finally, one of the objections is that SCDoc is not designed to do this. It is better to use other tools.

To which I somehow would somehow agree regarding to follow the unix philosophy, but I don't have the feeling that SC is pursuing this philosophy. Having SCDoc in the first place would be an example.

I think having Math support within SCDoc would be a good feature and I would vouch for it, especially as you are so passionate about it.

I yesterday went into the debian IRC and they said that most likely we have to modify the debian packaging - they suggested that we should get in contact with the package maintainer, which I will do today.
They also pointed out that some parts of MathJax violate the debian policy, see https://salsa.debian.org/js-team/mathjax/-/blob/debian/sid/debian/copyright.

---

I also tried out a small build with Temml yesterday with mixed results - it seems the used chromium engine (qt 5.15 is using chromium 87) is currently too old to render MathML properly (which is a pity) - it would have allowed to simply add 3 files which were together not exceeding 400kb. Simple things like inline \frac{2}{4} are rendered wrong and \int_0^\infty does not get rendered in display mode.

Compare firefox

vs Qt WebEngine

[smoge]

I am sorry to hear this and I can relate to your frustration. I still would really like and invite you to be part of participation as your input in discussion, code and motivation is really valuable and appreciated!

I just want to find a good modus operandi together to tackle "controversial" topics such as this b/c I believe that just maintaining a status quo will just lead to a decay of contributions.

Thank you for your response, @capital-G.

To be clear, I was not refering to my participation and contribution to the SC community/project. When I see sensible people, with knoledge and openness, I'm more than happy to do teamwork on easy, medium, or hard tasks.

I think my motivation to participate in this discussion, is because it's not a fair interchange as other situations. I see a lot of ideas circulating without arguments but imposed as dogmas (as if we were a Church). I'm not refering to you, or @mtmccrea. But some ex-developers in the last years created a "CREDO" and believe it's some kind of obvious "good practice in CS". But everyone knows that when you just repeat something without argumentation and debate, it's just a tautology, you're imposing something arbitrary.

Besides, the way those strange opposition (without real arguments) did not follow The Code of Conduct, which I reproduce a relevant passage:

--

The SuperCollider community strives to be:

• Friendly and patient.

• Welcoming: We strive to be a community that welcomes and supports people of all backgrounds and identities.

• Considerate: Your work will be used by other people, and you in turn will depend on the work of others. Any decision you take will affect users and colleagues, and you should take those consequences into account when making decisions. Remember that we’re a world-wide community, so you might not be communicating in someone else’s primary language.

• Respectful: Not all of us will agree all the time, but disagreement is no excuse for poor behavior and poor manners. We might all experience some frustration now and then, but we cannot allow that frustration to turn into a personal attack. It’s important to remember that a community where people feel uncomfortable or threatened is not a productive one.

• Understanding of differences of opinion: Disagreements, both social and technical, happen all the time. It is important that we resolve disagreements and differing views constructively. Remember that we’re different. The strength of our community comes from its diversity. Different people have different perspectives on issues. Being unable to understand why someone holds a viewpoint doesn’t mean that they’re wrong. Don’t forget that it is human to err and blaming each other doesn’t get us anywhere. Instead, focus on helping to resolve issues and learning from mistakes._

--

I don't see here as a place for confrontation.

I would much prefer to see Supercollider's GitHub a place of TUTORING, where knowledge circulates. A coder with less experience can grow much faster with senior developers pointing things here and there.

The practical result of those attitudes of a very small number of people is pushing the project more and more into conservatism, and eventually, it will just lose its relevance. I don't want to see this.

During the situation involving my name, I wrote a long letter about ethics in the SuperCollider community. I planned to make it public but to avoid more turbulence, I gave a copy only to the moderators.

But I still believe ethical introspection must happen in this community until it becomes a very different place.

[capital-G]

@smoge thanks for your text. I think we should dissect and re-visit (edit: the social dynamics and interactions of this) thread/PR at another time and space in order to find a more fluent experience for everyone with less friction. This is very important.

On the topic, as temml does not work, I tried to implement tex support via kaTeX, see

develop...capital-G:supercollider:scdoc-katex

kaTeX and MathJax have some different priorities and technical capabilities, but I think they are not too important here.

More important to me seems that kaTeX is much easier to build and bundle b/c it does not require to load modules on-demand. git clone and yarn build a sufficient to generate the 2 files (katex.min.css (21kb) and katex.min.js (275kb)) and a folder with 20 woff2 fonts with a size of 260kb, so a complete footprint of 563kb.
I think it is fair to include them as static builds b/c jquery and codemirror are also included as static builds and a build through yarn would only be necessary for an update of katex which is very likely not necessary within the next 5+ years.

Although there is a debian package for katex (https://packages.debian.org/sid/katex), the package is used to install katex as a CLI, therefore it is not applicable to the same/shared library approach within debian (at least to my understanding), eliminating this problem as well.

[smoge]

Suggestion @capital-G: new PR, we will have a small team to help you

One suggestion. I think your main expertise is probably HTML/JavaScript/web services. (Just a guess). Am I right?

All information is available here in this thread, including primary information from communications with MathJax developers. All problems were solved. Most of the unknowns are not problems anymore. Just some elements, such as Debian packaging, still need some conclusions.

I think you are a very good candidate to lead this project. You're at home in technical terms and can have a team with expertise in other things.

We can design a plan in no time with the information available.

Since you're already working with small changes in the SCDoc parser, that would be a good task to be coupled together anyway.

Again, if you are willing to lead a team like this, with respect among comrades, I would very glad to help if needed.

[scztt]

One suggestion regarding packaging: what if we package MathJax exactly as it's provided on CDNs (e.g. I believe a single, minified js file), and treat the copy in our sclang build as more or less a cached version of the CDN copy, for cases where a user either doesn't have internet, or chooses to use a local copy for other reasons (there can be an SCDoc flag for this).

We don't need to include MathJax at all in the SuperCollider repo, we can simply pull down the version we want via e.g. cmake file (https://cmake.org/cmake/help/latest/command/file.html) or FetchContent (https://cmake.org/cmake/help/latest/module/FetchContent.html), which could (?) solve debian problems. If this way of pre-caching mathjax is somehow still against Debian policy, this approach would still easily allow us to simply turn off "cached local MathJax" on Debian and instead use an external like - this would mean, on Debian only, math equations would only render if you're online, which seems like an acceptable limitation if it avoids a packaging nightmare.
It would also make it easy for us to flexibly choose to either use local or CDN MathJax (e.g. with a SCDoc renderer option, or even an HTML option that falls back to local in case of no-internet).

[prko]

@capital-G

Thank you for this summary! Maybe edit the top post to include it so others are able to catch up with the current state?

Done!

---

@capital-G

More important to me seems that kaTeX is much easier to build and bundle b/c it does not require to load modules on-demand. git clone and yarn build a sufficient to generate the 2 files (katex.min.css (21kb) and katex.min.js (275kb)) and a folder with 20 woff2 fonts with a size of 260kb, so a complete footprint of 563kb.

@smoge already mentioned it, and I told him that as far as I know, MathJax's result is better than KaTeX's...

However,

• I do not know the details because I have not compared the two tools. To know which tool is better, we need to test each tool with complex formulas.
• we did not know about the problem with Debian policy.... at the time,

Anyway, I think the Debian policy problem is currently the most important and the last problem to be solved. Including a script that installs the MathJax package in HelpSource/lib is the last thing I can do... I am not an expertise on this, so I cannot judge whether this would be good or not.

If you want to continue with KaTeX, I can close this PR,

If someone implements the following

• inline math for plain text
• inline math for teletype text
• display (block) math

I will be happy to use it and I will be satisfied that my engagement has moved an expert mind.

---

@capital-G

An additional point for consideration: If math/tex support breaks, how much else breaks?

Please compare the images below (one of them shows how the page looks like when the MathJax library is missing):

[scztt]

Re mathjax caching, the cmake is relatively easy:

if(CACHE_LOCAL_MATHJAX)
    include(FetchContent)
    FetchContent_Declare(
        mathjax
        URL https://cdn.jsdelivr.net/npm/mathjax@3.0.0/es5/tex-mml-chtml.js
        DOWNLOAD_NO_EXTRACT TRUE
    )
    FetchContent_MakeAvailable(mathjax)
    message(INFO "MathJax downloaded to ${mathjax_SOURCE_DIR}")
endif()

As well as including this in the files that are copied during install, along with our various .css files etc.
Note that the same approach would also work for KaTeX - the work could also be done in this PR, and shouldn't even differ that much, no?

Re the mathjax vs katex choice, Id suggest that a big priority here should be long-term stability - our documentation doesn't get updated much, and the SCDoc also doesn't get updated much (for good reason! these are the kinds of things you do right once and then leave alone, hopefully). One concern with Katex is that it's not yet 1.0 - it's explicitly using semantic versioning, which means we have no real guarantee of API stability at all.

I guess it's worth considering what it would look like if, 1 or 2 years from now, Katex changes their API significantly, which would mean we would need to rewrite any Javascript code that touches it. This may not be a big deal, maybe we barely need to touch the library, but I think the "what would it take to update" is a really important thought experiment.

[capital-G]

As mentioned above, I think kaTeX has the advantage of a smaller footprint, cleaner and simpler packaging, no problems with Debian packaging, and supposedly better performance than MathJax, although it doesn't cover 100% of the Latex functionality - but I don't think we'll need more than 5% of Latex for our needs.

Are there any advantages of mathjax over kaTeX that could be relevant? Otherwise I'd suggest to switch to kaTeX in order to have an easier process of integrating this feature.

@prko

If someone implements the following
...

• inline math for teletype text

This would change the SCDoc syntax as

teletype::
foo STRONG::bar::
::

does not render foo bar but

so I don't think we should add an exception here for a potential math context.

---

@scztt

Re mathjax caching, the cmake is relatively easy:

if(CACHE_LOCAL_MATHJAX)
    include(FetchContent)
    FetchContent_Declare(
        mathjax
        URL https://cdn.jsdelivr.net/npm/mathjax@3.0.0/es5/tex-mml-chtml.js
        DOWNLOAD_NO_EXTRACT TRUE
    )
    FetchContent_MakeAvailable(mathjax)
    message(INFO "MathJax downloaded to ${mathjax_SOURCE_DIR}")
endif()

IIRC this mathjax.js loads lots of modules and fonts async via JS so this would still be online-only. Also I'd prefer to make a full build possible without the need for an online connection (beyond package manager and source code download).

As well as including this in the files that are copied during install, along with our various .css files etc. Note that the same approach would also work for KaTeX - the work could also be done in this PR, and shouldn't even differ that much, no?

It's 99% the same but with less and smaller files, easier build and w/o the debian issues.

Re the mathjax vs katex choice, Id suggest that a big priority here should be long-term stability - our documentation doesn't get updated much, and the SCDoc also doesn't get updated much (for good reason! these are the kinds of things you do right once and then leave alone, hopefully). One concern with Katex is that it's not yet 1.0 - it's explicitly using semantic versioning, which means we have no real guarantee of API stability at all.

I guess it's worth considering what it would look like if, 1 or 2 years from now, Katex changes their API significantly, which would mean we would need to rewrite any Javascript code that touches it. This may not be a big deal, maybe we barely need to touch the library, but I think the "what would it take to update" is a really important thought experiment.

MathJax is on transition between v2 (debian), v3 (the current one) and a beta v4 (the next one).
At the same time once JS runs it rarely gets deprecated. The JS code to render the formula is minimal (10LOC) and I don't think it's a problem in case the API of MathJax or kaTeX changes at some point - but if we don't update this wouldn't change at all.

[smoge]

Are there any advantages of mathjax over kaTeX that could be relevant? Otherwise I'd suggest to switch to kaTeX in order to have an easier process of integrating this feature.

Only advanced DSP, or some plugins for block signal notation or something like this, which we would not import anyway.

Let's start with the simplest solution. The only thing I think is important is the LaTex synthax. Let's stick to it whatever library.

[smoge]

Block diagrams (MathJax) seem helpful if Supercollider and Faust achieve better integration in the future.

[prko]

@capital-G

In the post (#6260 (comment)), an image was not included correctly and has been fixed. Please compare the two images again to see what happens when the MahJax package is missing or broken.

---

@capital-G

If someone implements the following
...

inline math for teletype text
This would change the SCDoc syntax as

teletype::
foo STRONG::bar::
::
does not render foo bar but

foo STRONG:: bar::
so I don't think we should add an exception here for a potential math context.

In this PR, LaTeX code can be used in teletype without using SCDoc's math:: modal tag, thanks to the following line in the SCRenderer.sc file:

https://github.com/prko/supercollider/blob/18348356dfa3709ff0b7b13ab0fbc53a8652bb44/SCClassLibrary/SCDoc/SCDocRenderer.sc#L255
<< "<script>window.MathJax = { options: { skipHtmlTags: ['script', 'textarea'] } }</script>\n"

Does KaTeX provide such a script?

---

KaTex is known for faster rendering than MathJax, but KaTex does not render all LaTeX code.
There is a web page showing rendering performance and rendering details, including qualities of KaTeX, MathJax 2.7 and MathJax 3.2:

https://www.intmath.com/cg5/katex-mathjax-comparison.php

There is some code that KaTeX does not render (arrows and symbols):

The quality of both is not that different, and the speed is KaTex does not seem to be really faster than MathJax on my end, if I understand the table entitled Timings.

---

@scztt

Thanks for your response! I think your suggestion seems to resolve all the problems. However, according to @capital-G, your solution seems to be based on online access to MathJax server. Then, I think some developers who pointed out that the MathMax package should be shipped with SC would not accept it even though it is limited to Debian... Do I understand it correctly? As my knowledge regarding this is not enough, I should rely on the interpretation of other developers. I am sorry for that.

---

@smoge
I understand how you feel and think and thank you for your commitment to this PR!

[prko]

Um...
How long I spent researching this is not important to the development of SC.

As far as I am concerned, the decision to close this PR and wait for @capital-G's PR is not up to me, but that of the other developers.

As I am not a professional developer, I do not know all the underlying issues. So if @capital-G still writes that KaTex is better than MathJax for SC after reading this post, I should believe him until some other developer has an opinion that supports this PR with enough convincing reasons.

---

Please keep in mind the following four things:

1. The power of KaTex and MathJax
   (including the graphing feature mentioned by @smoge)
   MathJax > KaTex (bigger is better)
2. The size of the library:
   MathJax > KaTex (smaller is better)
   • MathJax:
     • extracted: 7.4 MB
   • KaTex:
     • extracted: 2.3 MB
3. Debian policy
   • MathJax
     1. Proposal of this PR:
        1. When building SC, this PR does not copy the MathJax 3.2.2 package into the `HelpSource/lib' subdirectory of the build directory.
           <- already done
        2. Instead, it installs MathJax 3.2.2 using npm in the `HelpSource/lib' subdirectory of the build directory.
           <- should be done when this PR moves forward.
        3. There is no need to modify the current SCDRenderer.sc file.
     2. Suggestion from @scztt:
        Use a cached version.
        <- Done
   • KaTex
     There is no problem.
4. Rendering quality
   Both are press quality.

---

My verdict:
At the moment I do not know if I need the functionalities that MathJax has and KaTex does not. But it is better to have a more competent tool.

---

What is your verdict?

[prko]

Oh, I did `rebase' the branch linked to this PR to reflect the new changes from the merged commits, as this is the recommended way to do a branch update. Um... this seems to delete all the commit history. I cannot see what I have been doing for a month.....

[prko]

I am closing this PR since @capital-G suggested #6317.

I think MathJax is better than KaTeX. As the two images in #6260 (comment) show, KaTeX cannot do what MathJax can do.

I think the way to do it for Debian policy in this PR does not cause any problems, but as I am not an expert in this area, I follow the opinion of @capital-G.

Thanks to @capital-G for implementing KaTeX, and I am glad that my involvement at least got you to implement the math feature for SCDoc!

[capital-G]

@prko

I think MathJax is better than KaTeX. As the two images in #6260 (comment) show, KaTeX cannot do what MathJax can do.

MathJax has some more features than kaTeX, but I really think that these are not really common or not part of the LaTeX standard at all, like the mentioned schematic diagram.

For a full list of available functions see https://katex.org/docs/support_table

The graphic could suggest that evergreens such as Rightarrow are not supported, but in fact all the "failed" examples render properly in kaTeX, see

On the other hand, the output of kaTeX actually matches the typesetting of LaTeX, so you also have features in kaTeX that you don't have in mathJaX.

Thanks to @capital-G for implementing KaTeX, and I am glad that my involvement at least got you to implement the math feature for SCDoc!

I am really grateful for your dedication to implementing this feature, which was the reason I sat down and tried it once again kaTeX which I think has more chances against the mentioned opposition. Personally, I am not too interested in TeX support, although I support adding this feature, and thought I could use my expertise to get this in.
The PR with kaTeX is more of a "restart" of this PR, and it would not have happened without the discussions in this PR, because we all learned something.

I hope I didn't create any bad feelings by creating the 2nd PR :/

[prko]

@capital-G

I just wanted to say a big thank you for the extra info. With this info, I'm fully happy now! I hope your PR gets the green light as soon as possible!

Your hard work, effort and time for the needs of all users are much appreciated!