Overhaul code highlighting #278

Totto16 · 2024-04-20T21:14:56Z

Overhaul code highlighting

Description

As described in #271 code highlighting should become optional, so that user need to install an additional package or something similar to make it work.

There are a few options, I'll test all of them and list benefits from them and shortcomings

Overview

We need an option that can do these things:

detect languages, say how "confident" (highlight.js calls it relevance) the guess is, based on content only (no file extension or similar things, or shebangs)
format languages to Pango markup or something similar, that we can convert to it
provide custom styles or use a set of predefined styles or be easy to parse / and or add custom styles to it
support a few languages, not just 1,2

Our options are:

1: Use a command-line highlighter, that comes with many distributions, it can be installed and is then detected by Pano, to use it
2: use a JavaScript package as before, the user needs to install files into a correct location, and then Pano can detect it.
3: make a custom deb / rpm / arch etc. package with Pano and built-in highlighting, that installs the needed package automatically

1. Option

Pros:

the packages are quite robust and have many languages
it is easy to install for the enduser.

Cons:

we need to "shell out" e.g. spawn a shell process, that is more complex and error-prone than other methods

Feature Table

package name	detection	many languages	markdown output	styleable
source-highlight	❌	✔️	⚠️	⚠️
pygments	⚠️	✔️	✔️	✔️

Notes

source-highlight:

It can not detect languages without filename or shebang, it also can't tell you, how confident it was (there is a option to infer the language, but it only works on file extensions / shebangs or similar metadata)
It can't output to markdown, but to simple html, which would be styleable in some way
You can specify custom css or similar things, but it's not that easy, but maybe doable.

pygments(python3-pygments)

It is good at detecting creating languages (like python) but when I tried JS, or HTML or some other ones it failed ... i didn't test 100 files, but it wasn't that good, the one good thing was, that it's better than highlight.js in many cases and the detected languages all seem to look similar, when highlighted.
It has support for many many languages, it is easy styleable and can output pango markdown out of the box.

update: after some modification to the detection logic, it seems really stable and robust when working with more than ~ +- 200 characters. The shortcoming of it is, that it has no real relevance value, but we can emulate that, by using that charaacter length. After that it is really good.

2. Option

Pros:

it is easy to integrate
it has many options
many highlighter have many languages and are quite good

Cons:

If we don't ship them, it's not easy for the end-user to install

3. Option

Pros:

We can pre package everything we need, the cons of the 2. Option or the previous implementation go away
We can select option 1 or 2 as core implementation

Cons:

We would stop publishing on EGO or the package published there is not complete and maybe, depending on the use option (1 or 2) it might be not easy to add the code highlighting to Pano again by hand

Current state

I would focus on option 1, since it's the best choice atm. I tested source-highlight but it's quite bad. So I'll be using pygments(python3-pygments) for now.

TODOs:

fix all bugs with the new implementation
fix slider (Gtk4.Scale) Size in Settings
more testing
fix search of styles in settings

optional:

Improve pano theme for pygments

Type of change

Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Checklist

My code follows the style guidelines of this project
My commits follow the commit standards of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have checked my code and corrected any misspellings

Totto16 · 2024-04-20T22:29:54Z

@oae Do you have an auto branch protection rule in some way, if the branch isn't named feat/ (is that even possible)
Since I am currently unable to push (it fails with the reason, that this is a protected branch ❓)

oae · 2024-04-20T23:06:54Z

The first option with pygments sounds good to me too

oae · 2024-04-20T23:25:04Z

@oae Do you have an auto branch protection rule in some way, if the branch isn't named feat/ (is that even possible) Since I am currently unable to push (it fails with the reason, that this is a protected branch ❓)

I have changed some settings now. Can you try again?

Totto16 · 2024-04-20T23:37:46Z

The first option with pygments sounds good to me too

@oae Do you have an auto branch protection rule in some way, if the branch isn't named feat/ (is that even possible) Since I am currently unable to push (it fails with the reason, that this is a protected branch ❓)

I have changed some settings now. Can you try again?

Same error, strange, it seems, that I can't push to any branch on this repo, but I could earlier this day 🤔

Edit: indeed, it is fixed ❤️

oae · 2024-04-20T23:45:41Z

I think it should be fixed now

…js references

- use interface that permits multiple highlighters in the future

- fix many settings errors, so that they behave correctly - use setter of `sensitive` instead of `set_sensitive` - correctly connect settings and the core `_markdownDetector` of the extension - display code items as text items, if there is no code highlighter, or he is disabled

- change paths, that now require async routes in the code flow - add LoadingPanoItem, that shows up, before an item is loaded

…another way: make an not highlighted version of the code item, and after the highlighting returns, set the correct markdown - type metaData of all items explicitly with new TS wrapper methods to JSOn.parse and JSON.stringify

- always catch errors - add load status to PangoMarkdown

- add mor logger types and use them - add a slider(Gtk4.Scale) to set the relevance threshold for the highlighter - fix Promisified types - add some more inline comments about (not needed atm) not implemented features - chnage CodeItem handling of later initialization

…etector is loaded

…nous scanning - fix createDropdownRowForHighlighter, it had some errors in using some outer scope variables incorrectly

- fix scale issue, when resetting - fix small issue in panoItemFactor - refactor how teh style in codePanoItem is handled, it now also listens to text events and can mimic text style, only if it's set to code it display the code

… the beginning and when changing theme it doesn't create 100 of shell requests in one thread, this might slow down the gnome-shell, so distributing it is better

- use it by setting PYTHONPATH env variable to the correct place - installing it into the dist folder (might be a bad idea)

… environment only)

… way to build it - add Pipenv, to install poetry, for the build step - change the directory, where it's locally installed (it's optional now) - set the PYTHONPATH nevertheless, since if it's not there, it doesn't matter, later we can offer the wheel for users to install globally, so that it's opt-in (we porblay can also publish it to pypi?

- change the PYTHONPATH location in the code

…is intended like that)

- pygments styles, add more colors and bump version

- add more sophisticated scheduled job handling: now they can be cancelled, which is done at disabling and also, if we disable the code highlighter, so that no unnecessary tasks are run

…hlighter gets it's options at the start from teh settings - add SpinRow for the Highlighter settings - add relevanceLength setting to pygments, taht replaces a hardcoded value

- completely reinstall the dependencies, to get newer sub-dependencies

HearseDev · 2024-05-31T01:26:24Z

Not really experienced with anything gnome, but was wondering if this would be a viable and much cleaner solution. It could be an optional lib install and also has bindings for js. Or it could be required and could be used as a default text view?(think it might be preinstalled with gnome, not too sure) I am just throwing out an idea, never really worked with anything remotely close to this.
https://wiki.gnome.org/Projects/GtkSourceView
https://gnome.pages.gitlab.gnome.org/gtksourceview/gtksourceview5/class.PrintCompositor.html

Totto16 · 2024-05-31T14:51:36Z

Not really experienced with anything gnome, but was wondering if this would be a viable and much cleaner solution. It could be an optional lib install and also has bindings for js. Or it could be required and could be used as a default text view?(think it might be preinstalled with gnome, not too sure) I am just throwing out an idea, never really worked with anything remotely close to this. https://wiki.gnome.org/Projects/GtkSourceView https://gnome.pages.gitlab.gnome.org/gtksourceview/gtksourceview5/class.PrintCompositor.html

I think I saw this in my research, but It isn't half as good as pygments in autodetecting languages and it's ballpark is nearly the same as source-highlight.

It has integrated GTK support, but it's not themeable (at least easily) and it has no "real" autodetection feature, you have to create a widget to get the language, which is not suitable for our purpose 😓

Totto16 mentioned this pull request Apr 20, 2024

Issue tracker for Full GNOME 46 support #271

Open

4 tasks

Totto16 added 4 commits April 21, 2024 01:52

remove highlight.js

9b988fe

remove prismjs, as it is not needed anymore (atm)

dd931d0

implement code highlighting interface,, remove prismjs and highlight.…

900bf0d

…js references

fix errors an make highlighting work as intended, when it is disabled

141b7d7

Totto16 force-pushed the overhaul-code-highlighting branch from 3c71dc7 to 141b7d7 Compare April 20, 2024 23:53

implement basic code highlighting with pygments

3001330

- use interface that permits multiple highlighters in the future

Totto16 force-pushed the overhaul-code-highlighting branch from 925b926 to 3001330 Compare April 21, 2024 13:20

Totto16 added 17 commits April 21, 2024 21:04

WIP: add settings for code highlighter

ca8008d

improve eslint config

0795f3e

use async subprocess calls

6b00265

- fix all eslint errors /warnings

16e634a

- change paths, that now require async routes in the code flow - add LoadingPanoItem, that shows up, before an item is loaded

fix updating of codePanoItem, so that it works as expected

a25220b

fix a few async errors:

063b7c2

- always catch errors - add load status to PangoMarkdown

format eslintrc

b32b4e1

use newer "date-fns" package

51e7817

update rollup

0e2c70e

many fixes:

d918114

- add mor logger types and use them - add a slider(Gtk4.Scale) to set the relevance threshold for the highlighter - fix Promisified types - add some more inline comments about (not needed atm) not implemented features - chnage CodeItem handling of later initialization

fix initial creation of code items, it now waits, until the markdownD…

a54068c

…etector is loaded

fix scale row

a16078c

- fix log spam in case of disabled highlighter

b6fae07

remove scan method and use onLoad of markdownDetector to get asynchro…

a09bf75

…nous scanning - fix createDropdownRowForHighlighter, it had some errors in using some outer scope variables incorrectly

fix a few errors:

4da0816

- fix scale issue, when resetting - fix small issue in panoItemFactor - refactor how teh style in codePanoItem is handled, it now also listens to text events and can mimic text style, only if it's set to code it display the code

Totto16 added 9 commits April 23, 2024 20:49

- add mechanism to schedule concurrent markupCode request, so that in…

b8d4b1f

… the beginning and when changing theme it doesn't create 100 of shell requests in one thread, this might slow down the gnome-shell, so distributing it is better

add custom pygments style

4dd3050

use poetry and better python pygments style config

ad19779

- completely add the python styles

3f3a74f

- use it by setting PYTHONPATH env variable to the correct place - installing it into the dist folder (might be a bad idea)

minor dependency updates

fff612a

add a note about the validate-color package for the future

61f7dbb

catch exception, that might occur (it's really rare, and in the debug…

e14d433

… environment only)

- also test the pygments style plugin build in teh CI

28f7afe

- change the PYTHONPATH location in the code

Totto16 force-pushed the overhaul-code-highlighting branch from 578ba28 to 28f7afe Compare April 23, 2024 19:30

Totto16 added 6 commits April 23, 2024 21:34

- prettier: ignore python code

c5055ca

- add some void annotation to potential promises, that we ignore (it …

2b71f34

…is intended like that)

- python ci, try to fix it

02c4107

- pygments styles, add more colors and bump version

- sort style values of highlighter

5bc5f7f

- add more sophisticated scheduled job handling: now they can be cancelled, which is done at disabling and also, if we disable the code highlighter, so that no unnecessary tasks are run

use GLib.timeout_add instead of setTimeout

e74f760

fix ci

ed68ae5

Totto16 force-pushed the overhaul-code-highlighting branch from 2d3b82d to ed68ae5 Compare April 23, 2024 21:54

Totto16 added 2 commits April 24, 2024 01:08

- refactor constructor of PangoMarkdown, so that each constructed Hig…

290a04a

…hlighter gets it's options at the start from teh settings - add SpinRow for the Highlighter settings - add relevanceLength setting to pygments, taht replaces a hardcoded value

fix small dumb mistake

1c79443

This was referenced Apr 24, 2024

Settings Preview #279

Open

Extension Outdated #267

Open

Totto16 added 2 commits May 5, 2024 19:09

bump dependencies

3039d98

- update @Girs type dependencies

a5c7bf4

- completely reinstall the dependencies, to get newer sub-dependencies

Totto16 mentioned this pull request May 8, 2024

Overhaul UI #280

Open

9 tasks

Totto16 added the Priority High Priority High label May 8, 2024

fix property name change in gnome-shell types

26dbcd6

Totto16 mentioned this pull request May 8, 2024

Add further steps to Development ReadMe #281

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Overhaul code highlighting #278

Overhaul code highlighting #278

Totto16 commented Apr 20, 2024 •

edited

Totto16 commented Apr 20, 2024

oae commented Apr 20, 2024

oae commented Apr 20, 2024

Totto16 commented Apr 20, 2024 •

edited

oae commented Apr 20, 2024

HearseDev commented May 31, 2024 •

edited

Totto16 commented May 31, 2024

Overhaul code highlighting #278

Are you sure you want to change the base?

Overhaul code highlighting #278

Conversation

Totto16 commented Apr 20, 2024 • edited

Overhaul code highlighting

Description

Overview

1. Option

Feature Table

Notes

2. Option

3. Option

Current state

Type of change

Checklist

Totto16 commented Apr 20, 2024

oae commented Apr 20, 2024

oae commented Apr 20, 2024

Totto16 commented Apr 20, 2024 • edited

oae commented Apr 20, 2024

HearseDev commented May 31, 2024 • edited

Totto16 commented May 31, 2024

Totto16 commented Apr 20, 2024 •

edited

Totto16 commented Apr 20, 2024 •

edited

HearseDev commented May 31, 2024 •

edited