py-editor lacking pyscript module

[WebReflection]

It's been brought up that it's desired to eventually import pyscript module within a py-editor and that's not currently the case.

Hooks are super clean on py-editor but it means import pyscript fails big time.

Apparently with latest setup feature users are happy anyway, but I think it's reasonable to expect our own namespace/module out of the editor our own code creates.

This requires some investigation and it's not as trivial as it sounds, but if everyone else agrees that's a nice thing to do, I'd be happy to dig further and make it happen.

---

Agreed rule of thumb in short

• did you specify an env ? be sure that, if you ever need a config there, the first editor with that env has it. If you have a setup editor tag, be sure that config is in there before any other editor with the same env lands on the page.
• did you not specify an env ? all good, if you have a config that editor will use it without issues.

---

MR #2010

[ntoll]

For the sake of posterity... x-posting from our discord channel, and hoping to nurture some discussion here, and so we can reach a concordance about how things are expected to work:

@JeffersGlass said:

Maybe I'm way out of touch here, but I can't seem to import the pyscript package in a py-editor?

<script src="mini-coi.js"></script>
<script type="module" src="https://pyscript.net/releases/2024.3.2/core.js"></script>
<script type="py-editor" env="one">
import pyscript
</script>

ModuleNotFoundError: No module named 'pyscript'

(In the py-editor)

import os
os.listdir(".")

[]

@WebReflection responded:

in short it's always been the case ... editors are 100% clean envs, nothing we provide on terminal or regular scripts is in there because everything is different (i.e. error handling, where to display things which is part of the editor, and so on). We can keep it simple or make it very complicated, it's up to us.
@JeffersGlass now I am also going to ask you why you need pyscript module in there ... to do what? we need to be careful in what we enable because we have already a target output, the editor as input, and errors handled within the output as required.
happy to learn about use cases, so far two devs got surprised, but beside import pyscript I haven't seen any concrete use case ... even the document is complicated in there to be honest because they might mess up the editor, but also editors are always in workers, which adds complexity.

My 2c is... as @WebReflection mentions in the OP, making the pyscript namespace available is a not-trivial task since the scope/context of py-editor is special because of the "clean environment" situation and Las Vegas like what-happens-in-the-editor-stays-in-the-editor outlook, which is both desirable and makes the implementation simpler (and thus less prone to bugs and easier to support). However, I think we can all agree that from pyscript import fetch is likely to be a common requirement for users of the editor, along with other aspects of the pyscript namespace. I'm not 100% certain what my position is yet -- I'd like to investigate and think further -- but my intuition is that we should allow the pyscript namespace but with a lot of caveats and clear warnings in our docs. It feels like we're handing them a bazooka and saying "don't point it at your foot and pull the trigger". On their head be it, etc...

I'd love to hear more from @WebReflection (#webyoda) about the possible technical approaches, pros/cons etc we might take to move forward on this.

Ultimately (and I think resonating with Jeff), I'd like to use the py-editor features in blog posts and articles so they're also an interactive coding environment. A case in point: I'm currently writing a "from the ground up" blog post/explanation of how neural networks and LLMs work. Guess what the example code is written in..? 😉

As always folks, thoughts, ideas and constructive critique are most welcome and gratefully received. 🚀

[WebReflection]

The py-editor is special because, unless a setup attribute is used to bootstrap the thing, its code never runs unless explicitly executed via the Run button.

It doesn't define pyscript hooks because it has its own to orchestrate stdio and other things differently. I might try to bring only pyscript in but I am worried when we then add features or change something in there we might break editors use cases.

In particular:

• the display is broken/misleading in py-editor because the logic is completely different as the output is directly "displayed" in the related editor target ... we might override it in there but it looks ugly as that's a pyscript module field, so it's not trivial to do so and it would break display expectations to me
• the current_target is also different, hence broken or misleading
• the sync is awkward because py-editor only works in workers, so there's no way to use in any meaningful way
• RUNNING_IN_WORKER would always be true, pretty useless
• the PyWorker would be awkward
• we've left the document, window, because import js already works out of the box, fetch and js_modules but imho there's not much else to offer ... also ... what about PyDom?
• the document and window needs SharedArrayBuffer to work which IIRC is currently not a requirement for the py-editor, even if it runs in workers ... but I need to double check this

The PyEditor is basically using raw XWorker from polyscript and it's enabled by PyScript but it's like a thing a part that would work 100% without needing PyScript at all.

I am not saying this is what we want or what users expect, I am saying this is the current state and how it landed and worked to date.

edit strawberry on top, the PyEditor has zero knowledge about config too so that js_modules won't work in there out of the box ... the setup field was meant to solve all of this manually bringing in only files or dependencies that are needed but I understand this is not super user friendly although bringing the config in means delaying or parsing configs a part per each env and because the main feature of the PyEditor is the shared, still unique, environments, by name or interpreter, adding configs on the DOM will be cumbersome and definitively these should not interfere with PyScript configs and it's very hard to explain to me, unless we allow a config attribute and we branch its parsing ... that's the cleanest thing I can think about but no <py-config> for PyEditor should land, imho.

[WebReflection]

P.S. after all, the type is py-editor or mpy-editor, not just py or mpy, the editor is a custom polyscript type itself and it could be as well be a project a part (but it's not convenient).

[WebReflection]

what I think would be reasonable and hopefully not too complicated: #1997 (comment)

[WebReflection]

edit folks, you know what? after all this "faffing around" (in Nicholas words 🤣) it looks like I can bring in once the whole stdlib as that's already exposed by core and it requires 3 LOC changes ... I think my concerns might be premature and I rather have concrete failing cases than re-implementing a branching logic for the sake of few utilities needed in the py-editor itself.

The only missing bit would be then to implement the config for the setup env script only, and I don't see any reason why we shouldn't do just that to allow everything else as usual in pyscript ... if we agree on the setup only parsed config, and I can throw errors if config is present in non setup nodes, I think we'll be good.

This way I've found doesn't require any special treatment to anything and it bootstraps our stdlib once and never again, it feels like a win-win to me.

I am leaving in here my previous comment but it looks like nothing in there, except the setup idea, is a concrete concern at all, or something difficult to do.

---

My after thoughts: this is to me the cleanest solution I can come up with:

• the config attribute is parsed only on setup ... this avoids having a config per each shared env, promotes the usage of the setup per env and never clashes with previous config (i.e. files that land and override over and over previous files already landed and packages not present before and so on)
• the pyscript module will provide only the following:
  • ffi
  • fetch
  • window
  • document
  • js_modules only when specified in the related config for the editor env in a setup marked tag

<script type="mpy-editor" config="./mpy.toml" setup></script>
<script type="mpy-editor">
  from pyscript.js_modules import thing
  from pyscript import ffi, fetch, window, document
  # ... users' code in here ...
</script>

<!-- same with a specific env name -->
<script type="mpy-editor" config="./mpy.toml" env="task1" setup></script>
<script type="mpy-editor" env="task1">
  # ... users' code in here ...
</script>

If this is agreed I would be happy to find a way to bring those things in ... after all, beside the ffi which is a specific Python thing, everything else should be already available via polyscript the same way is available in PyScript and for the fetch case I have a 100% fallback so that ffi is literally the only thing that requires some orchestration in the build step but it can work.

If we instead want the whole pyscript namespace there's much more to consider and test, but I am not strictly opposed to it, it just doesn't feel right in most occasions and I don't think editors need the whole module exposed API ... but maybe I am wrong here.

[JeffersGlass]

For context - I ran across the desire for the using the pyscript module in editors when I revived an older project, which uses AlpineJS to reactively render infomation about Python objects. The code in question (not working) is on GitHub - you can see my goal was to use set.settrace() and frame.f_trace to automatically update alpine's knowledge of the objects in global scope whenever Python enters/exits a frame.

I think your proposed solution @WebReflection would work well for me - having access to pyscript.document, pyscript.window, and pyscript.js_modules (conditionally) woud I think suffice for all my cases. I'm also grabbing the Event class from JS and throwing a custom Event, but I should be able to grab that through pyscript.window.

At this point, I don't see the need to add the entirety of the pyscript namespace... I could see an argument for having @when, not so much for users, but for using that inside editors in our own documentation. But that's a pretty meta use-case, and I don't feel strongly about it.

[WebReflection]

uhm ... I've updated my proposal as things turned out to be way easier than expected: #1997 (comment)

[WebReflection]

@ntoll @JeffersGlass I think we can have the cake and eat it too #2010

the port was actually way easier than anticipated because I forgot we already have the whole pyscript module served as string and enough architecture to just bootstrap that string anywhere else we like/want to ... so if we agree on the config landing only on setup tags I think we're already close to make everyone happy, including myself, around this overly reasonable feature request.

Bear in mind my concerns around display and the rest still stand, but I think we can live and see / fix as opposite of overly-complicating our life already for something that maybe won't ever happen ... and this in favor of consistency for what we ship as module so it feels like a win-win to me.

[WebReflection]

OK, my latest PR does the following (copy and paste of the MR details):

• use a codeBeforeRun hook that executes on XWorker bootstrap once and includes our stdlib artifact, as that's exactly what any other PyScript tags uses anyway
• add a check for config attribute, and, if there:
  • if no env was specified, use that config
  • if an env was specified and known:
    • if the env had a config, throw an duplicated config for env: ...
    • if the env had no config, throw an unable to add a config to the env: ...
  • bootstrap the worker for that env with that config otherwise
• flag the env as known regardless
• test everything works as expected

This logic addresses all your expectations @ntoll so we should be good to go now.

[WebReflection]

to be fully clear about what's landing, your last 2 NO in your example throws different errors:

throws: duplicated config for env: test

<script type="mpy-editor" env="test" setup config="config.json">
a = 1
</script>
<script type="mpy-editor" env="test"  config="config2.json">
print(a)
</script>

throws: unable to add a config to the env: test

<script type="mpy-editor" env="test" setup>
a = 1
</script>
<script type="mpy-editor" env="test" config="config.json">
print(a)
</script>

the setup at this point is a side effect ... if you setup an env and you don't specify the config in there, any other same env with a config will throw the latest error so it's implicitly required, for a setup to have both env and config, whenever latter is needed.

I think this reasons well ... the branching logic is both not too convoluted and it has meaningful errors to explain, and all use cases with YES will be allowed.

As soon as reviewed, I'll also publish it on npm so we can all play around with it.

---

edit rule of thumb in short:

• did you specify an env ? be sure that, if you ever need a config there, the first editor with that env has it. If you have a setup editor tag, be sure that config is in there before any other editor with the same env lands on the page.
• did you not specify an env ? all good, if you have a config that editor will use it without issues.

@ntoll @JeffersGlass do we all agree on this simplification?

---

[WebReflection]

P.S. I am still thinking if the setup with no env should dictate the default config to all non-env editors, but I feel like this can be iterated later on, if our users believes that should happen ... a setup without an env is indeed a bit awkward right now, as it's not clear why that should be allowed (it is, and it might do other things, like a DOM setup before other editors, who knows!).

[JeffersGlass]

Seat of my pants, I would be confused if all editors without an env shared the same config as specified by setup. It clashes with my casual mental model of “editors with no env are isolated and alone”

From a usability standpoint I see the value, lots of isolated editors that are behaving similarly and therefore want to share a config, but I think the solution in that case should be telling users to just point all those editors at the same config file, which would accomplish the same thing.

[WebReflection]

fair enough, but that's nowhere in the MR anyway 😁

[WebReflection]

This discussion can be closed as the feature landed via #2010