Allow control of lexer used for syntax highlighting

[SeanDS]

I use jupyter-sphinx to show examples using Python to execute an application-specific language:

.. jupyter-execute:

    # Parse a simple model.
    model.parse(
        """
        l l1 P=10
        s s1 l1.p1 m1.p1
        m m1 R=0.995 T=1-&m1.R
        s s2 m1.p2 m2.p1 L=10
        m m2 R=0.9999 T=1-&m2.R
        pd pd1 m2.p1.i
        pd pd2 m2.p2.o
        """
    )

This gets rendered like this:

I've got a pygments syntax highlighter that can handle highlighting our special syntax inside the string, inside Python code. I would like to change the lexer type used by Sphinx for the cell that gets compiled. This would make it render like this:

Currently the lexer used by jupyter-sphinx is hard-coded here. Is it possible to change this on a per-cell basis?

[akhmerov]

AFAIR the lexer isn't hardcoded, the line you refer to is just providing the custom ipython lexer to sphinx.

jupyter-sphinx infers the lexer from kernel metadata. Therefore, I think, you need to register a custom ipython kernel (kernels already have a pygments lexer metadata entry) that executes your code, and then also add your lexer to sphinx.

Does that sound like a reasonable approach?

[philj56]

Hi @akhmerov, I'm on the same team as @SeanDS. Thanks for the help! I've got things working locally doing it this way, but there seemed to be a few issues in jupyter-sphinx that stop it from working at first.

Firstly, the language attribute is set on the wrong node in execute.py, as it currently sets the language of the CellInputNode, rather than the child literal-block . This means this line isn't actually doing anything (which you can confirm by commenting out that logic or setting lexer to whatever you like - the resultant div in html output always has a class of highlight-default). Instead, it should be:

source = node.children[0].children[0]

Fixing this then causes syntax highlighting to break, as the line here defines the wrong name - it should be "ipython3" (as per the definition in the ipython package)

After fixing both of these the highlighting works as intended, and for example in the html output, normal jupyter-execute blocks now have a class of highlight-ipython3 rather than the current highlight-default, and our custom highlighting works if I change the kernel.

Hopefully this makes sense and I've understood what's going on. I can open a pull request if you want, but it's only a two line change. Thanks again for the help!

[akhmerov]

@philj56 thanks for following through! Your description sounds correct, and I'd really appreciate a PR with the fixes.

[philj56]

No problem, it's #181

[akhmerov]

@philj56 @SeanDS I think this issue may probably be closed now, is that right?

[SeanDS]

Looks like it! Thanks.

EDIT: any idea when the next release will be? Keen to use @philj56's fix!

[akhmerov]

I would like to include a few more things in the 0.4 milestone (#165 seems pretty important, also #169 should be useful).