Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

please provide a default nonblocking async input driver #204

Open
pmp-p opened this issue Nov 21, 2023 · 21 comments
Open

please provide a default nonblocking async input driver #204

pmp-p opened this issue Nov 21, 2023 · 21 comments
Assignees
Labels
enhancement New feature or request

Comments

@pmp-p
Copy link

pmp-p commented Nov 21, 2023

eg based on select.select + os.read ( works fine already with nurses2/batgrl and Textual

for cases where sys.platform in ('emscripten','wasi') ( but not pyodide)

eg typical render task could look like :

import asyncio, select, os

if sys.platform in ("emscripten", "wasi"):
    async def main():
        # save cursor in case alt screen fails
        STDIN = sys.stdin.fileno()

        loop = asyncio.get_event_loop()
        try:
            while not loop.is_closed():
                await asyncio.sleep(0)  # or 0.016 if not using RequestAnimationFrame
                if select.select([STDIN], [], [], 0)[0]:
                    try:
                        payload = os.read(STDIN, 1024)
                        # await handle terminal stream, process events
                    except OSError:
                        continue

                # save cursor
                # render damaged screen zone.
                # restore cursor

        except asyncio.exceptions.CancelledError:
            print("Cancelled")
        except KeyboardInterrupt:
            print("KeyboardInterrupt")
        finally:
            pass
            # restore term, stop mouse reporting, restore cursor
@ceccopierangiolieugenio
Copy link
Owner

ceccopierangiolieugenio commented Nov 21, 2023

hi, is this issue related to pyTermTk?
if you need some help I already overcome this issue with pyodide using a proxy. (i.e. Sandbox)
or you can use the pyTermTk input driver.

If this is related to pyTermTk, (i.e. trying to run it under pygame-web)
are you able to provide an example or the instructions to reproduce the issue?
I didn't spent much time on it but It is not clear how to run a custom python project using wasi/Emscriptem.

@ceccopierangiolieugenio
Copy link
Owner

ceccopierangiolieugenio commented Nov 21, 2023

I tried your input routine using pygbag
but I cannot get any keypresses from stdin
how is the stdin/out handled?
I amprinting in the stdout a vt100 ansi sequence, is your default screen compatible?
did textual, ncurses2 worked out of the box?

@ceccopierangiolieugenio ceccopierangiolieugenio added the enhancement New feature or request label Nov 21, 2023
@pmp-p
Copy link
Author

pmp-p commented Nov 21, 2023

sorry i did not think you would try directly on pygbag - i'm honoured - but the select.select patch is still brewing on my pc and is triggered for now by the first termios.tcsetattr() call that is expected to switch the terminal in raw mode. it's very crude
but does the job until patching emscripten is worthwhile.

        __select = select.select

        def patch_select(rlist, wlist, xlist, timeout=None, /):
            global __select
            # stdin
            if rlist[0] == 0:
                return [embed.stdin_select()]
            return __select(rlist, wlist, xlist, timeout)

        select.select = patch_select

        def patch_os_read(fd, sz):
            return embed.os_read()

        os.read = patch_os_read

the embed module from main.c manage to store all keypressed/mouse events in a c-string that happen beetween two frame call ( either from xterm.js or from a replay ). stdin_select just returns that c-string lentgh while os_read returns a suitable pyobject and truncate the c-string at the same time preventing further read until next frame

fixing emscripten select would allow pyodide to behave normally regarding TTYs. it's not a problem to implement tty ioctl on js side since both pyodide-micropython can all access js it is just a matter of having enough people that agree on how to do it.

For now i'm dumping everything that can/should be fixed here https://github.com/pygame-web/platform_wasm/blob/main/platform_wasm/todo.py you are very welcome to give input, it has been years i did not touch ansi

nb: but in the end it is not specific to web, and should work embedded in async any program too at least on posix OSes.

@pmp-p
Copy link
Author

pmp-p commented Jan 25, 2024

Please just provide a simple normal event loop, ie without threads and too much OOP :

async def async_mainloop():
   while True:
       ttydata = ""  # os.read(fd)
       process_events(ttydata)
       paint_term()
       await asyncio.sleep(0)

asyncio.run( async_mainloop() )

Not only it would solve wasm/wasi but should also serve for offscreen rendering and input replay tests.

@ceccopierangiolieugenio
Copy link
Owner

ceccopierangiolieugenio commented Jan 25, 2024

what you mean about without too much OOP?
The entire pyTermTk design and structure is based on OOP
My library is already compatible with pyodide that is a Wasm impllementation of CPython.
If you can show me an example or the steps you are trying to run on your framework I can try to investigate a solution,
i.e. With a driver for your use case.
I cannot rewrite the core routine just on those hints if I cannot test it.

@pmp-p
Copy link
Author

pmp-p commented Jan 25, 2024

My library is already compatible with pyodide that is a Wasm impllementation of CPython.

I disagree you are using pyodide proxies : they are not part of cpython. They are a one of possible implementations of interacting with a js host and there are others bridges of that kind and other compilers than emscripten.

So i'm just hoping for a very simple and generic event loop that :

  • take a stream of utf-8 in input and make events from it
  • process events if any/draw full buffer on init.
  • output a stream of utf-8 as output.

Leaving up to the platform to implementing read and write on a real (or not) local (or not) tty.

Your general oop toolkit design is very clean, but imho oop/threading add no value to the core event loop itself because they hide too much of the I/O control flow and its implementation for outside viewer.

i don't have a framework at all, i only use cpython-wasm with four file descriptors instead of the standard three ( in short ALTBUF has its own non blocking fd for input )

The select+os.read above is just common mockup for classic unix terminal handling which seems to suit other terminal toolkit but that should not even be required for a serial terminal (tx/rx no ioctl).

regarding pyodide although i'm a contributor i am (was?) only interested in the module build system and the cpython emscripten support (which has been dropped for cpython 3.13 this month).

I have no intention of diving in pyodide specifics and no one should have to when sticking to POSIX ( async, no threads, standards FD).

i don't think you have to rewrite anything , just document the pyodide (driver input / render step / driver output) control flow so a simpler driver (eg based on async or requestAnimationFrame ) can be derived from it.

@ceccopierangiolieugenio
Copy link
Owner

Are you able to provide me an interface with those input/output stream I can test?
The stdin/out used in Linux/Windows/Pyodide are different, for this reason I had to create a driver for each and a proxy for pyodide since I was not able to find an out of the box solution.

About pyodide,
its support for threading and I guess also asyncio is limited/not working,
Inits implementation,

I used a single thread and any extra TTkThreading objects are emulated through the built-in Javascript Timeout routines.

I had also to rewrite the input routine since it doesn't provide a proper tty device.
So, any input stream coming from xterm.js is pushed straight to the input handler of pyTermTk
and any push to the stdout is instead sent to the xterm.js
there is no threading involved.

Any keypress is captured by:

        this.ttk_input  = this.namespace.get("ttk_input");
        this.term.onData((d, evt) => { this.ttk_input(d) })

and sent to pyTermKk through:

            def ttk_input(val):
              kevt,mevt,paste = TTkInput.key_process(val)
              if kevt or mevt:
                TTkInput.inputEvent.emit(kevt, mevt)
              if paste:
                TTkInput.pasteEvent.emit(paste)

It is not ideal because it skip an extra threaded routine I use to clear the input buffer, but for my needed was ok.

@pmp-p
Copy link
Author

pmp-p commented Jan 26, 2024

cpython-wasm behaves the same as linux/mac/posix, the same async code is valid on desktop and also supports generators based greenthreads

code:
https://github.com/pygame-web/showroom/blob/main/src/test_vt100.py

test:
https://pygame-web.github.io/showroom/pythongit.html#src/test_vt100.py

@ceccopierangiolieugenio
Copy link
Owner

ceccopierangiolieugenio commented Jan 26, 2024

Are you able to provide me some instructions I can use to try my library on your framework?
If I can detect that I am running on your environment I can make a driver optimised for it
(as it happen for pyodide or windows),
I don't want to change the linux core routine.
How are you handling the tty/stdin/out?

@pmp-p
Copy link
Author

pmp-p commented Jan 26, 2024

again, there's no framework it's normal posix cpython stdin is handled here https://github.com/pygame-web/showroom/blob/c0a07cb8a9bf8416eb31937c4e4e12dab1b090f1/src/test_vt100.py#L115

you don't have a linux or mac to run that code ?
if not try WSL2

@ceccopierangiolieugenio
Copy link
Owner

I am using linux.

@pmp-p
Copy link
Author

pmp-p commented Jan 26, 2024

then the same code will work.

@ceccopierangiolieugenio
Copy link
Owner

no, because it already runs flawlessly on linux and I don't want to change the core routine.
but if I can detect that I am running on wasm I can make a driver.

@ceccopierangiolieugenio
Copy link
Owner

I noticed that when I was working on pyodide the os type was emscriptem
is it the same for you?

@pmp-p
Copy link
Author

pmp-p commented Jan 26, 2024

the normal way to detect wasi or emscripten is testing sys.platform
it is important to handle both right now, as wasi is the only official wasm platform for cpython git

@ceccopierangiolieugenio
Copy link
Owner

it is ok, as long I can detect it I can make the driver
because the input routine won't be the only one that is going to change, there are many blocking threads and timers all around the code.
how can I run my demo on your environment?

@pmp-p
Copy link
Author

pmp-p commented Jan 26, 2024

python3 -m pip install pygbag
name your demo main.py next to the https://github.com/ceccopierangiolieugenio/pyTermTk/tree/main/TermTk folder and you can use python3 -m pygbag --ume_block 0 main.py or python3 -m pygbag --PYBUILD 3.12 --git --ume_block 0 main.py for more recent and it will package main.py + pyTermTk for the browser and provide a xterm console and then you can go to http://localhost:8000?-i

@pmp-p
Copy link
Author

pmp-p commented Jan 26, 2024

beware of blocking threads they can lock up browser tabs for good if you ever forget to add a yield on vsync.

@ceccopierangiolieugenio
Copy link
Owner

I made a simple python script I wanted to try:

import platform

def main():
    print("Eugenio")
    print(f"Platform {platform.system()=}")

if __name__ == "__main__":
    main()

if I try to run it using:

python -m pygbag --PYBUILD 3.12 --git --ume_block 0 main.py

I get this error:

self.path='/pygbag/0.0/cpython312/main.wasm' path='/home/one/github/Varie/pyTermTk/tmp/build/web/pygbag/0.0/cpython312/main.wasm'                                                                                                                                                                                            
ERROR 404: https://pygame-web.github.io/pygbag/0.0/pythonrc.py                    
127.0.0.1 - - [26/Jan/2024 17:32:54] code 404, message File not found  
127.0.0.1 - - [26/Jan/2024 17:32:54] "GET //pygbag/0.0/pythonrc.py HTTP/1.1" 404 -                                                                                     
ERROR 404: https://pygame-web.github.io/pygbag/0.0/cpython312/main.wasm                                                                                                
127.0.0.1 - - [26/Jan/2024 17:33:18] "GET //pygbag/0.0/cpython312/main.wasm HTTP/1.1" 200 -                                                                                                                                                                                                                                  
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 58968)        
Traceback (most recent call last):                                                 
  File "/usr/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)           
  File "/usr/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)                                                                                                   
  File "/usr/lib/python3.10/http/server.py", line 668, in __init__ 
    super().__init__(*args, **kwargs)
  File "/usr/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()            
  File "/usr/lib/python3.10/http/server.py", line 433, in handle            
    self.handle_one_request()
  File "/usr/lib/python3.10/http/server.py", line 421, in handle_one_request                                                                                           
    method()            
  File "/home/one/.venv.python/lib/python3.10/site-packages/pygbag/testserver.py", line 73, in do_GET    
    f = self.send_head()      
  File "/home/one/.venv.python/lib/python3.10/site-packages/pygbag/testserver.py", line 143, in send_head                                                                                                                                                                                                                    
    with h_cache.open() as fh:                                                     
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open                                                                                                            
    return self._accessor.open(self, mode, buffering, encoding, errors,                                                                                       
FileNotFoundError: [Errno 2] No such file or directory: '/home/one/github/Varie/pyTermTk/tmp/build/web-cache/e9594fa69936d600b1002b61e034ff4f.head'                                                                                                                                                                          
----------------------------------------                                       
                                                                                                                                                                       
                                                                                   
self.path='/pygbag/0.0/cpython312/main.wasm' path='/home/one/github/Varie/pyTermTk/tmp/build/web/pygbag/0.0/cpython312/main.wasm'
ERROR 404: https://pygame-web.github.io/pygbag/0.0/cpython312/main.wasm                                                                                       
127.0.0.1 - - [26/Jan/2024 17:33:25] "GET //pygbag/0.0/cpython312/main.wasm HTTP/1.1" 200 -                                                                                                                                                                                                                                  
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 58384)        
Traceback (most recent call last):                                                 
  File "/usr/lib/python3.10/socketserver.py", line 683, in process_request_thread
    self.finish_request(request, client_address)           
  File "/usr/lib/python3.10/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)                                                                                                   
  File "/usr/lib/python3.10/http/server.py", line 668, in __init__ 
    super().__init__(*args, **kwargs)
  File "/usr/lib/python3.10/socketserver.py", line 747, in __init__
    self.handle()                                                              
  File "/usr/lib/python3.10/http/server.py", line 433, in handle                                                                                              
    self.handle_one_request()                                                  
  File "/usr/lib/python3.10/http/server.py", line 421, in handle_one_request                                                                                  
    method()                                                                   
  File "/home/one/.venv.python/lib/python3.10/site-packages/pygbag/testserver.py", line 73, in do_GET                                                                                                                                                                                                                        
    f = self.send_head()                                                       
  File "/home/one/.venv.python/lib/python3.10/site-packages/pygbag/testserver.py", line 143, in send_head                                                                                                                                                                                                                    
    with h_cache.open() as fh:                                                 
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open                                                                                                   
    return self._accessor.open(self, mode, buffering, encoding, errors,                                                                                       
FileNotFoundError: [Errno 2] No such file or directory: '/home/one/github/Varie/pyTermTk/tmp/build/web-cache/e9594fa69936d600b1002b61e034ff4f.head'                                                                                                                                                                          
----------------------------------------   

@pmp-p
Copy link
Author

pmp-p commented Jan 26, 2024

@ceccopierangiolieugenio
Copy link
Owner

yes, now it works, the network I was using is not very stable.
Thanks.
I will check this soon, I've a busy WE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants