[WIP] [Draft] Upgrade to Deno 1.30.3 #463

DavidDeSimone · 2023-02-12T18:17:58Z

This PR is a work in progress (WIP) PR to upgrade to Deno 1.30.3. This PR will attempt to maintain backwards compatibility while adding new APIs.

This PR changes the way we implement deno. Previously, deno was run on the same thread as the lisp VM. This enabled a high degree of interoperability, but cause some performance issues. Specifically, we had to manually pump the deno event loop via a lisp timer - a process that was very fragile. Certain deno operations will only do so much work within a single cycle of the event loop. You wouldn't notice this in deno mainline because deno normally has full control over it's own event loop. Taking that control from deno and putting it under the lisp VM has a large impact on our ability to async load files.

Now, we will have 3 threads:

The main lisp thread
The deno thread
A lisp worker thread

When deno wishes to execute a lisp function, it will be querying a lisp worker thread. This allows us to solve a problem that ng-js had in calling lisp -> js -> lisp where the main thread is waiting on the js thread, and the js thread is waiting on the main thread.

This will make our new APIs all async/future based. My goal will be that calling

(js-eval-string-async "...") will return a 'future' that you can get the result of via
(js-resolve future)

Our previously sync functions will just automatically resolve the future for you - keeping old behavior.

On the JS side, the lisp.function style functions now become async and will require

await lisp.function().

Goals

My goals are to improve performance, and to set up the stage to be able to use this code across different platforms. emacs-ng JS functionality still cannot be used on windows - my long term solution for that is to allow this code to also be used as an emacs module, and to allow dynamic linking on windows to solve that issue.

Remaining Work (will be updated over time)

Finish hooking up lisp worker -> js flow
Adding 'future' functionality for non-blocking JS tasks
Add JS/TS region/buffer functions
Handle lisp proxies
~~- Add lisp function proxies (May be cut due to difficulty)~~

DavidDeSimone · 2023-02-12T18:18:19Z

Cargo.in

@@ -22,7 +22,7 @@ name = "emacsng"
 description = "Experimental emacs fork"
 license = "GPL-3.0"
 version = "0.1.0"
-edition = "2018"


A requirement of deno

DavidDeSimone · 2023-02-12T18:18:46Z

rust_src/crates/js/Cargo.toml

@@ -14,13 +14,18 @@ lsp_json = { version = "0.1.0", path = "../lsp_json" }
 libc = "0.2.95"
 lazy_static = "1.2"
 futures = "0.3"
+deno = { git = "https://github.com/DavidDeSimone/deno", branch = "emacs-ng@1.30.3" }


A finalized version of the PR will use emacs-ng's deno fork vice my own

Updating git repository https://github.com/DavidDeSimone/deno

This line takes forever like before. Maybe we should find a better way?

Let me hack around on this and see what I can do.

@declantsien I made some changes for the deno submodule to reduce clone size - can you let me know if it is still taking a long time for initial clone for you?

DavidDeSimone · 2023-02-19T05:16:23Z

I'm anticipating about 1-2 weeks from today (18FEB) until I have this in a polished usable state - from there I can build on top of it and iterate further. Currently, you can evaluate javascript and get the result to lisp, and you can call lisp -> js -> lisp. I need a better degree of auto marshalling, along with stripping color console characters from the deno output.

DavidDeSimone · 2023-03-01T07:03:12Z

Making good progress - the last three major items are:

Lower the download size (maybe pre-compiled, maybe strip the stuff we aren't using out of deno).
Proxies
Better error handling around the lisp -> js -> lisp flow.

DavidDeSimone · 2023-03-09T09:08:27Z

This patch is currently in a usable state - you can call JS from lisp, and query lisp from JS. The biggest remaining items are better error handling around JS (lips errors throwing a JS error when the promise resolves), and lisp proxies.

Proxies are what allows code like this:

const proxy = await lisp.currentBuffer();
const result = await lisp.bufferSize(proxy);

In that context, proxy is a special object that represents a lisp object.

Now, I do have eval'ing lisp from js working, so it may be that proxies are actually just increasing the complexity here more then they are worth. Instead, I could focus on an API that manipulates the world of the editor via eval or primitive calls.

declantsien · 2023-03-09T09:22:44Z

David DeSimone ***@***.***> writes:

@DavidDeSimone commented on this pull request. > @@ -14,13 +14,18 @@ lsp_json = { version = "0.1.0", path = "../lsp_json" } libc = "0.2.95" lazy_static = "1.2" futures = "0.3" +deno = { git = "https://github.com/DavidDeSimone/deno", branch = ***@***.***" } @declantsien I made some changes for the deno submodule to reduce clone size - can you let me know if it is still taking a long time for initial clone for you? -- Reply to this email directly or view it on GitHub: #463 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

Yeah, the download time is way fast now. But I am getting this error ld: alloc.o: in function `garbage_collect': /shared/src/WREmacs/src/alloc.c:6063: undefined reference to `Fjs__sweep' collect2: error: ld returned 1 exit status

DavidDeSimone · 2023-03-09T09:25:57Z

David DeSimone @.> writes:
@DavidDeSimone commented on this pull request. > @@ -14,13 +14,18 @@ lsp_json = { version = "0.1.0", path = "../lsp_json" } libc = "0.2.95" lazy_static = "1.2" futures = "0.3" +deno = { git = "https://github.com/DavidDeSimone/deno", branch = @." } @declantsien I made some changes for the deno submodule to reduce clone size - can you let me know if it is still taking a long time for initial clone for you? -- Reply to this email directly or view it on GitHub: #463 (comment) You are receiving this because you were mentioned. Message ID: @.***>
Yeah, the download time is way fast now. But I am getting this error ld: alloc.o: in function garbage_collect': /shared/src/WREmacs/src/alloc.c:6063: undefined reference to Fjs__sweep' collect2: error: ld returned 1 exit status

My bad, I was a little too aggressive in my deletions, should be fixed now.

declantsien · 2023-03-09T09:50:26Z

David DeSimone ***@***.***> writes:

> David DeSimone ***@***.***> writes: My bad, I was a little too aggressive in my deletions, should be fixed now. -- Reply to this email directly or view it on GitHub: #463 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

Well, it compiles and runs now. But I have no idea how to use it in Emacs. Need sometime to figure that now. Can't remember how this used to work. ```js const proxy = await lisp.currentBuffer(); const result = await lisp.bufferSize(proxy); console.log(result); ``` Where is the console.log output? Also I tested some typo, there is no error popup with `js-eval-buffer` ```js const proxy = await lisp.currentBuffer(); const result = await lisp.bufferSize(proxy); consoledsf.log(result); ``` I guess I am using it the wrong way. Does the js documentation needs updates?

DavidDeSimone · 2023-03-09T09:57:53Z

The docs are going to need an update - I will work on that as well. For now, the most basic usage will look like

(js-resolve (js-eval-string "console.log('hello js');"))

There is also (js-eval-file "filepath") for an entire JS file.

Once in JS, you call lisp via the special lisp object. It auto-magically translates from camelCase to kebab-case.

await lisp.print("hello");

console.log should log to term if you launch the GUI through term.

declantsien · 2023-03-09T10:16:51Z

David DeSimone ***@***.***> writes:

The docs are going to need an update - I will work on that as well. For now, the most basic usage will look like `(js-resolve (js-eval-string "console.log('hello js');")` console.log should log to term if you launch the GUI through term. -- Reply to this email directly or view it on GitHub: #463 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

Yeah, I saw some output. Not sure whether it is from js. Unknown parameter: ?2004 Unknown parameter: ?2004 Unknown parameter: ?2004 Unknown parameter: ?2004 For this `(js-resolve (js-eval-string "console.log('hello js');")`, I am have Emacs backtrace popuo now. Debugger entered--Lisp error: (tty text nil) js-resolve(nil) (progn (js-resolve (js-eval-string "console.log('hello js');"))) eval((progn (js-resolve (js-eval-string "console.log('hello js');"))) t) Nice work BTW.

DavidDeSimone · 2023-03-09T18:30:14Z

David DeSimone @.> writes:
The docs are going to need an update - I will work on that as well. For now, the most basic usage will look like (js-resolve (js-eval-string "console.log('hello js');") console.log should log to term if you launch the GUI through term. -- Reply to this email directly or view it on GitHub: #463 (comment) You are receiving this because you were mentioned. Message ID: @.>
Yeah, I saw some output. Not sure whether it is from js. Unknown parameter: ?2004 Unknown parameter: ?2004 Unknown parameter: ?2004 Unknown parameter: ?2004 For this (js-resolve (js-eval-string "console.log('hello js');"), I am have Emacs backtrace popuo now. Debugger entered--Lisp error: (tty text nil) js-resolve(nil) (progn (js-resolve (js-eval-string "console.log('hello js');"))) eval((progn (js-resolve (js-eval-string "console.log('hello js');"))) t) Nice work BTW.

I also forgot - you need to call (js-initialize) first to start the deno env. That won't be required in the final API, but is just part of my PR for now. though I haven't seen those errors you were seeing, they may or may not be related.

declantsien · 2023-03-10T01:25:23Z

David DeSimone ***@***.***> writes:

I also forgot - you need to call `(js-initialize)` first to start the deno env. That won't be in the final API, but is just part of my PR for now. though I haven't seen those errors you were seeing, they may or may not be related. -- Reply to this email directly or view it on GitHub: #463 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

I guess I just did thing the wrong way then. Will wait for the final API got stabilized. For your information `(js-initialize)` gives me this. Fatal error 6: Aborted Aborted

appetrosyan · 2023-10-01T09:26:32Z

Any chance we can help out with this?

DavidDeSimone · 2023-10-01T17:10:08Z

Any chance we can help out with this?

Hello,

I got a little distracted and never ended up finishing this, but it is in a pretty close to complete state. The biggest thing left was proxies, i.e. having a handle between JS objects and lisp objects. JS objects can have a hidden "value" field that isn't accessible directly by JS, but only the bindings. Our initial implementation stuck the lisp JS pointer as that data - though I think for this implementation having a "handle" (a hashmap key) into a map would likely be a better way, though the downside is that is requires a lookup each iteration isn't of just accessing the lisp object.

DavidDeSimone · 2023-11-17T17:11:52Z

Update on this - I think that pulling the JS VM into a separate thread ended up being a little too ambitious. I am going to try a new approach where I just upgrade deno to latest using the old VM/Sub-VM approach to get JS working again - from there, I think I can write a similar concept of the off-thread VM at the scripting layer.

DavidDeSimone added 3 commits February 11, 2023 15:28

WIP commit for getting deno up to 1.30.3

50184fc

Initial plumbing for lisp -> js -> lisp

d45117c

Referecing public repo for deno fork

36389bc

DavidDeSimone commented Feb 12, 2023

View reviewed changes

DavidDeSimone added 3 commits February 12, 2023 11:00

Additional work to get round trip values from lisp

c5491d4

Getting top level await working

e7fd617

Further iteration with inter-thread communication

e68b015

DavidDeSimone and others added 4 commits February 25, 2023 17:05

Adding ability to invoke with callback

6cb0cc1

Adding js eval file

3129422

Fixing up lisp error hangling for repl

c8dd78f

Updating callback logic, improving error handling.

bd7392b

Cleanup of javascript.rs, speedup of initial deno clone.

efa20fd

Re-adding js__sweep

f72a657

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] [Draft] Upgrade to Deno 1.30.3 #463

[WIP] [Draft] Upgrade to Deno 1.30.3 #463

DavidDeSimone commented Feb 12, 2023 •

edited

DavidDeSimone Feb 12, 2023

DavidDeSimone Feb 12, 2023

declantsien Feb 26, 2023

DavidDeSimone Feb 26, 2023

DavidDeSimone Mar 9, 2023

DavidDeSimone commented Feb 19, 2023

DavidDeSimone commented Mar 1, 2023

DavidDeSimone commented Mar 9, 2023

declantsien commented Mar 9, 2023 via email

DavidDeSimone commented Mar 9, 2023

declantsien commented Mar 9, 2023 via email

DavidDeSimone commented Mar 9, 2023 •

edited

declantsien commented Mar 9, 2023 via email

DavidDeSimone commented Mar 9, 2023 •

edited

declantsien commented Mar 10, 2023 via email

appetrosyan commented Oct 1, 2023

DavidDeSimone commented Oct 1, 2023

DavidDeSimone commented Nov 17, 2023

[WIP] [Draft] Upgrade to Deno 1.30.3 #463

Are you sure you want to change the base?

[WIP] [Draft] Upgrade to Deno 1.30.3 #463

Conversation

DavidDeSimone commented Feb 12, 2023 • edited

Goals

Remaining Work (will be updated over time)

DavidDeSimone Feb 12, 2023

Choose a reason for hiding this comment

DavidDeSimone Feb 12, 2023

Choose a reason for hiding this comment

declantsien Feb 26, 2023

Choose a reason for hiding this comment

DavidDeSimone Feb 26, 2023

Choose a reason for hiding this comment

DavidDeSimone Mar 9, 2023

Choose a reason for hiding this comment

DavidDeSimone commented Feb 19, 2023

DavidDeSimone commented Mar 1, 2023

DavidDeSimone commented Mar 9, 2023

declantsien commented Mar 9, 2023 via email

DavidDeSimone commented Mar 9, 2023

declantsien commented Mar 9, 2023 via email

DavidDeSimone commented Mar 9, 2023 • edited

declantsien commented Mar 9, 2023 via email

DavidDeSimone commented Mar 9, 2023 • edited

declantsien commented Mar 10, 2023 via email

appetrosyan commented Oct 1, 2023

DavidDeSimone commented Oct 1, 2023

DavidDeSimone commented Nov 17, 2023

DavidDeSimone commented Feb 12, 2023 •

edited

DavidDeSimone commented Mar 9, 2023 •

edited

DavidDeSimone commented Mar 9, 2023 •

edited