Syntax Highlighting

[ChrisPenner]

How can we standardize a useful representation of syntax highlighting usable across filetypes?

[ChrisPenner]

@clojj
As far as storing extension state I've found that if you embed your state in a data or newtype with named fields then you can write lens helpers that access them directly quite easily (as though they were part of the editor state). See rasa-ext-vim for an example.

[clojj]

ok, will take a look at it

[clojj]

btw, thinking about syntax highlighting Haskell...
Maybe people should know that (on OSX 10.11) you can change the default font of Terminal.app to Fira Code, which gives you ligatures !
(maybe something for the README.md ?)

https://github.com/tonsky/FiraCode
(does not work in iTerm.app !)

[ChrisPenner]

Nifty! I haven't seen that font before; I'm glad it just works like that!

[clojj]

embed your state in a data or newtype with named fields then you can write lens helpers that access them directly quite easily (as though they were part of the editor state)

works like a charm !
https://github.com/clojj/rasa/blob/master/rasa-example-config/app/Main.hs#L55

(hope I did it the right way... am somewhat new to the Lens library)

[clojj]

I have been thinking about tokenization.
Ignoring any optimizations on a lower level for now, I can see these general strategies:

a) keep tokenization completely in sync with the event loop, running after each buffer change
This is obviously simpler to implement. Lexer speed has to be fast enough.. I am pretty confident that it is in my case. Anything above pure tokenization (think parsing) will strain the eventloop though.
Also, it is not possible to have some delay here, so that multiple keypresses are "de-duped" (like in strategy c) )

b) make it an async Action, like you suggested previously. This is tricky to get right because of possible async changes to the text buffer after tokenization finishes. Performance could be suboptimal because of 'double-checking'

c) modify Rasa's eventloop, so that it controls the triggering of tokenization itself. For example, if there is no keypress after a configurable amount of time (say 300 ms), a 'timer' will trigger and call a registered tokenizer function. This call should be synchronous, so that any other keypress events will be dispatched after the tokenizer is finished.

...possible other strategies?

Caching and other optimizations can help in all cases for sure.
But I think it would be best to get the general strategy right first.. ?

[ChrisPenner]

Okay I took a look over your current implementation; the shell looks good; I believe it's possible to get around the use of explicit MVars in the final implementation if you use doAsync; Here's a shell of my initial idea (hasn't been tested):

data SyntaxHighlighter = SyntaxHighlighter 
  { _isProcessing' :: Bool
  }
makeLenses ''SyntaxHighligher

isProcessing :: HasBuffer b => Lens' b Bool
isProcessing = bufExt.isProcessing'

instance Default SyntaxHighlighter where
  def = SyntaxHighlighter False

highlighter :: Action ()
highlighter = onBufferChanged startParse

startParse :: Action ()
startParse bufRef = bufDo bufRef $ do
  processing <- use isProcessing
  -- If there's already lexing taking place don't spawn another job.
  unless processing $ do
    isProcessing .= True
    txt <- use text
    -- The next part we can do asynchronously
    liftAction $ doAsync $ asyncLex bufRef txt -- Run 'asyncLex' asyncronously

-- Everything in this function is done async; it returns the Action *describing* the next SYNCHRONOUS action which will
-- be run *eventually*
asyncLex :: BufRef -> YiString -> IO (Action ())
asyncLex bufRef oldText = do
  tokens <- lexText txt -- This is the slow part; we don't need forkIO; doAsync handles that.
  return (bufDo bufRef $ applyTokens txt tokens)

-- Everything in this function will *eventually* get pulled into the event loop and will be run synchronously thus
-- applying any changes. Things may have changed since 'asyncLex' ran.
applyTokens :: YiString -> [Token] -> BufAction ()
applyTokens oldText tokens = do
  -- Set the styles even if the text is outdated; what we have is probably better than what's currently set anyways.
  setStyles tokens
  newText <- use text
  isProcessing .= False
  -- If the text when we're finished is different than the text when we started; start over again.
  when (newText /= oldText) (liftAction startParse)

Alternatively; it would certainly be possible to build in some way to 'cancel' async jobs (since we just use Control.Concurrent.Async for those) if that would help. With the example I provided we're guaranteed to keep making progress even if the user keeps typing though; worst case the highlighting gets a bit behind; but that's still much better than freezing the editor.

The proper way to do this is incremental parsing; which incidentally some folks working on Yi have written a research paper on. Maybe we can integrate some of that knowledge as we go! I still have to give it a proper read yet.

---

As an alternative to triggering via keypress we could build in an activity 'debouncer' like you suggest without too much difficulty. Roughly speaking we could have Rasa dispatch an Inactivity event after some amount of time (preferably user configurable). Then the parser could listen for this event and trigger startParse in response.

Does any of that help??

Thanks for crunching away on this!

[clojj]

Ok, I'll safe your suggestion definitly as a 'backup solution'.
But I think I like your second idea better (at least for the moment)...
Can we have a Inactivity event ?

Actually I'm trying to get such an event in IO in my current experiments, it would sure make more sense to have this as a regular Rasa event. It'd be fantastic if you could prepare a branch for that (?)

[ChrisPenner]

Yup; I can whip that up in the next few days 👍

[clojj]

btw, congrats !

[ChrisPenner]

Oh wow! cool stuff! It's a team effort 🤜 🤛 👌

Also @clojj I noticed you're not in the Gitter Chat; you're missing a few interesting conversations there; consider joining us 😁

[clojj]

Ok, I have to look into your doAsync proposal... it may have good performance.
Also I'd like try debounce the incoming onBufferChanged events.
So I will take a shot at #20 first I guess.

As for using MVars.. they are needed because there is a loop running inside runGhc.
If you have any ideas here, please let me know !

About the other strategy (Inactivity event)...
Maybe Inactivity should guarantee that no buffer changes have occurred when its listener(s)are finished (?)

[clojj]

Two things...

1. initially I would like to focus on the Haskell lexer
   if that works out, a general lexing interface can be extracted/created

2. this looks very promising for debouncing BufTextChanged events:
   https://github.com/debug-ito/fold-debounce/blob/master/eg/synopsis.hs
   ... it eventuell folds the debounced values... in our case the buffer text-changes

[clojj]

example with fold-debounce:

  trigger <- Fdeb.new Fdeb.Args { Fdeb.cb = putStrLn, Fdeb.fold = (++), Fdeb.init = "" }
                      Fdeb.def { Fdeb.delay = 500000 }
  let send' = Fdeb.send trigger
  send' "a"
  send' "bc"
  send' "!"
  threadDelay 1000000 -- prints "abc!"
  send' "."
  threadDelay 1000    -- Nothing is printed.
  send' "."           -- prints ".."
  threadDelay 1000000 
  Fdeb.close trigger

...works as designed!

[ChrisPenner]

That's pretty cool; It looks like it'd be tricky to use the fold operator in our current event-loop/state-monad; however it would be easy enough to implement something like this using Effects in the Pipes.Concurrent lib by just adding a debounce pipe into the chain; I've been putting off migrating the event system to Pipes until necessary; but it's a well used library and can probably help us out there.

[clojj]

This is in Linux, running rasa inside a QTerminal (should run on all linux distros) with Fira Code and ligatures.

Thanks for pulling #20 ...hopefully I can make progress on lexing soon.