We should stop using pyodide.ffi, maybe?

[WebReflection]

We currently have PyWeb (or PyDom) broken in MicroPython and the reason is almost always the same: we use specific pyodide APIs that we don't get to control neither in Pyodide space nor MicroPython.

This is, imho, pretty bad for cross-portability of our own code but it's also ugly to me that anyone in MicroPython needs to import stuff from pyodide.ffi + various APIs or tricks needed in Pyodide are not really needed in MicroPython (or not reflected).

Currenty we are using:

• pyodide.ffi.to_js which is nowhere in MicroPython but most of the time not even needed
• pyodide.ffi.JsProxy which is also nowhere in MicroPython but that couples identities/references to a namespace they don't necessarily belong to neither and it's all a pyodide implementation detail our users shouldn't really care about
• pyodide.wrappers.add_event_listener which is also nowhere in MicroPython but also apparently never needed and, if the create_proxy = "auto" lands in polyscript, it won't ever be needed even in pyodide

I am here suggesting we provide a to_js and a utility to check if a reference comes from the JS world, such as is_js(ref) that would abstract away the need to ever import anything from pyodide namespace and will make both our code and our users code easier to reason about, as all they do is to import utilities from pyscript module, not random names leaked in other runtimes.

What do you think?

/cc @ntoll @fpliger @antocuni

[ntoll]

My 2c. TL;DR: Consistency is a good thing for folks using PyScript. Using unique workarounds depending on which Python interpreter you choose or need to use is not a great situation. It calcifies code and makes it brittle depending on the Python interpreter. It reminds me of the IE6-only problems we had in the mid-00s. I'm very much in the "write once, run everywhere" camp.

I hope our ongoing conversations about the FFI will help us move forward. We've got this. No matter how we proceed, our usual modus operandi of careful steps, respectful technical explorations and a collaborative frame of mind are key. We've got this, along with our Pyodide and MicroPython friends. cc/ @hoodmane @dpgeorge

[WebReflection]

Every interpreter already provides special own API, my worry here is that other interpreters might not need or look for full feature parity but even if we’d love to have that, they shouldn’t strictly follow pyodide namespace and naming convention.

the current use case for to_js is to convert otherwise Maps meant as JS object literals, MicroPython doesn’t do that by default, so do we need that at all for n there?

As the goal of the project is to abstract away complexity and make things as simple as possible, I think it’s also our duty to grant as much consistency as possible but not by simulating foreign namespaces when different constraints or implementations are around (and MicroPython is just one alternative to pyodide, we all know it won’t be the only one forever though).

I do believe we can have a respectful conversation and if anywhere in here I gave a different impression my apologies, I didn’t mean it, and I’m all for consistency, hopefully as much as anyone else involved in here.

[ntoll]

I do believe we can have a respectful conversation and if anywhere in here I gave a different impression my apologies, I didn’t mean it, and I’m all for consistency, hopefully as much as anyone else involved in here.

I'm not implying there wasn't respectful conversation - rather I just wanted to draw attention to the fact that we're VERY good at this sort of discussion and that I'm looking forward to making progress. 😃

[JeffersGlass]

I don't have concrete input to offer here on which functions we should replace/rename/shadow in the Pyodide FFI, but I'm +1 to having a layer of indirection between the PyScript "FFI" and the underlying interpreters. If there are functions that we really find useful enough that we're recommending users import from pyodide, we should shadow them in the PyScript module - which also givens us more flexibility to adapt them for different interpreters.

That said, then it's on us to provide documentation that is as-good-or-better than what's offered upstream.

[WebReflection]

as @antocuni suggested already, I fully agree when we need to explain, as an independent project, that to add a listener via DOM users should use instead pyodide.ffi.add_event_listener that, as well documented as it is, doesn't exist in MicroPython and it's not documented as Web standard, we're better off there providing a pyscript.add_listener or promote PyWeb instead for these kind of things, where everything can be normalized per each interpreter indeed ... so I fully support your comment and idea but also I think we shouldn't lower UX / DX because some project we depend on already documented that specific feature ... in this particular case, pyodide.ffi.add_event_listener is not specs compliant and doesn't allow a third parameter with possible options so to me, as Web standards oriented chap, that's rather more friction instead of a feature, and for features that represent well the current DOM standard API, we have tons of pre-written documentation out there, including MDN 😉

[fpliger]

Adding my +1 to adding an abstraction layer for all the reasons listed above. We may be able to remove some of that later (as @WebReflection showed recently, we can add some things to help quite transparently) but for now it's definitely needed.

I've had to do some of that internally when adding support for MicroPython in pydom.

[dpgeorge]

Also +1 from me to add an abstraction layer.

MicroPython could (and probably will) implement pyodide.ffi.to_js and pyodide.ffi.JsProxy, but it would be nice if it could be in a different namespace (and maybe even called a different thing), to match with MicroPython's API design. In this case an abstraction layer in PyScript is necessary.

[WebReflection]

it's happening here micropython/micropython#13583 (comment) and I can't wait for that hugely welcomed PR to land so we can actually tackle after all the details!

[mrinalraghupathi]

I'm no where near as expert in pyscript and web as the others on this thread, but I came across this issue as I was trying to figure out how to run some code (just learning how to use pyscript) in micropython after writing it in pyodide.

I am curious what is the current workaround for the lack of certain apis being availabe in micropython (if any). For example, I am using pyodifde.ffi.to_js to pass a dictionary as an object to a javascript function. Is there a way to do these things in micropython?

Also I notice that py-click needs to be changed to mpy-click. Is there a way to avoid this change?

[WebReflection]

1. there is work in progress in MicroPython about Python dictionaries to objects "out of the box" so you won't need that API or we could make it a no-op once that land, moving forward with the abstraction layer idea
2. py and mpy are 2 distinct worlds so that you need mpy-config and script type="mpy" and clicks can be also different per platform so you disambiguate via mpy-click but the HTML declarative approach is inferior to the @when decorator which is still the suggested way to add listeners, as that also happens rightly so when the the interpreter is ready and its code executed once

Specially about latest point, we did discuss this a lot last year and reached the conclusion that disambiguating is better than not disambiguating ... as both interpreters have a @when decorator (soon-ish to work in MicroPython too) and full access to the DOM API to also add listeners manually, we feel like we don't need to make interpreter related HTML events ambiguous ... and actually, if you want that you are 3 lines of code away from doing that automatically:

if (interpreter.type === 'mpy') {
  for (const el of document.querySelectorAll('[py-click]'))
    el.setAttribute('mpy-click', el.getAttribute('py-click'));
}

That's it.

[fpliger]

py and mpy are 2 distinct worlds so that you need mpy-config and script type="mpy"

+1 and I think that's something we can highlight more in our docs and examples to make this more explicit.

@when decorator (soon-ish to work in MicroPython too)

It should work in main right now