Add http middleware and websocket upgrade on existing listener #755

wilberforce · 2021-12-21T22:42:38Z

This PR makes the http webserver extensible by the addition of optional middleware. You can pick and choose what handlers you want to install or write new ones by subclassing Middleware.

Each handler is called in a chain until a URL match is found, if none found and 500 error is sent.

import { Server as HTTPServer } from "middleware/server";
import { MiddlewareHotspot } from "middleware/hotspot";
import { MiddlewareWebsocket } from "middleware/websocket";
import { MiddlewareRewriteSPA } from "middleware/rewritespa";
import { MiddlewareStaticZip  } from "middleware/staticzip";
import { MiddlewareStatus } from "middleware/status";

this.#http = new HTTPServer({
				port: 80
			}
		);
		//
		this.#ws = new MiddlewareWebsocket('/api');
		this.#http.use( this.#ws  )
		this.#http.use( new MiddlewareRewriteSPA( ['/failure','/success','/password','/choose','/connect'] ))
		this.#http.use( new MiddlewareStaticZip(this.#
[wifiprovision.zip](https://github.com/Moddable-OpenSource/moddable/files/7758733/wifiprovision.zip)
archive))
		this.#http.use( new MiddlewareHotspot())
		this.#http.use( new MiddlewareStatus())

The attached wifiprovision shows a complete example.

phoddie · 2021-12-22T22:03:49Z

@wilberforce – this looks very cool. I'm just starting to go thought it. My focus is the integration on the HTTP and WebSocket servers. If I'm reading it correctly, it looks this is what you do

When your module's HTTP server callback recognizes a path to the WebSocket server, it sets the status to 101
When the HTTP server sees a 101 status, it sends the response status line and then stops using the socket
Your module grabs the TCP socket from the HTTP server
Your module completes the WebSocket handshake using the socket
Your module turns the socket over to the WebSocket server to take care of sending and receiving messages.

Am I close?

wilberforce · 2021-12-22T22:34:47Z

@wilberforce – this looks very cool. I'm just starting to go thought it. My focus is the integration on the HTTP and WebSocket
Thanks!

Am I close?

Yes. Additionally the socket close is then handled by the Middleware

wilberforce · 2021-12-22T22:48:30Z

There is another event stream type that esphome uses for debug output and event status:

https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events

I think this would also be useful middleware, and thinks like console.log could be redirected/tee'd so that debug output could be shown in a page on the web server for debug output on wifi connected devices.

Here is a nodejs implementation server side:
https://stackoverflow.com/questions/36249684/simple-way-to-implement-server-sent-events-in-node-js

phoddie · 2021-12-23T00:41:30Z

It would be straightforward to implement server-sent events -- both client and server. I was never that motivated to implement it because WebSockets is is all that and more. But, I suppose for compatibility having server-sent events would be useful. But... that's another topic. This one is big enough already. Let's stay focused. ;)

phoddie · 2021-12-23T00:42:47Z

Yes. Additionally the socket close is then handled by the Middleware

I missed that, thanks. In my mental model, only one layer should own the socket at a time. Once the socket has been handed off to WebSockets, then WebSockets should be responsible for closing it. Another layer could take it back using detach. Is there a problem with that approach?

wilberforce · 2021-12-24T00:48:47Z

I missed that, thanks. In my mental model, only one layer should own the socket at a time. Once the socket has been handed off to WebSockets, then WebSockets should be responsible for closing it.

Yes - the websocket middleware takes responsibility

Another layer could take it back using detach. Is there a problem with that approach?

Not sure quite what you mean here?

phoddie · 2021-12-24T19:10:34Z

Only one layer/module should be responsible for the socket at a time. After it is handed off to the WebSocket server, no other module should hold a reference to it. That is the only way to be sure that no problems happen because of unsynchronized access by both.

Separately, the MiddlewareWebsocket class appears to duplicate the handshake process that the WebSocket server also contains. Is that for a functional reason or because it was difficult to re-use the existing implementation in place?

wilberforce · 2021-12-25T03:07:04Z

Only one layer/module should be responsible for the socket at a time. After it is handed off to the WebSocket server, no other module should hold a reference to it. That is the only way to be sure that no problems happen because of unsynchronized access by both.

This is how the connection is managed, once the socket has upgraded

moddable/modules/network/http/middleware/websocket.js

Line 154 in 7b60140

req.server.connections.push(websocket);

The connection is pushed into a list on the server object, so when the server is closed, so is the upgraded socket:

moddable/modules/network/http/middleware/server.js

Lines 67 to 69 in 7b60140

    
           close() { 
        
           	if ( this.connections ) 
        
           		super.close(); // Close any http connections

Separately, the MiddlewareWebsocket class appears to duplicate the handshake process that the WebSocket server also contains. Is that for a functional reason or because it was difficult to re-use the existing implementation in place?

Difficult to reuse. The http class already decoded all the headers so that code was not required. If the core parts of the switch were pulled out into methods then they could be re-used.

phoddie · 2021-12-26T23:52:20Z

Difficult to reuse

Understood. I tried anyway. The result works well and feels straightforward to use.

Let's walk through the client code. First, instantiate the servers. The HTTP server is instantiated as usual. SInce the WebSocket server will use connections that come in from HTTP, it doesn't need a listener socket. That is signaled by passing the constructor null for the port.

import {Server as HTTPServer} from "http"
import {Server as WebsocketsServer} from "websocket"

const websockets = new WebsocketsServer({port: null});
websockets.callback = function (message, value) {...};

const http = new HTTPServer({port: 80});

Then, the HTTP server callback needs to detect when a request is made to a Websockets endpoint (path). Here it just checks the path for /ws. To take ownership of the socket, I added detach to the HTTP server. That roughly parallels a detach that already exists in WebSockets. For the WebSocket server to take ownership of an existing socket, I added attach. It assumes that the socket has already processed the status line, but nothing more. With those additions in place, client HTTP callback is straightforward:

http.callback = function(message, value, etc)
{
	if ((HTTPServer.status === message) && ("/ws" === value)) {
		const socket = http.detach(this);
		websockets.attach(socket);
		return;
	}

	...
}

From there, no more HTTP callbacks are received on the detached connection and the WebSocket callback operates as usual. The WebSockets attach method doesn't invoke the Server.connect callback because it didn't accept the incoming connection.

Time is a little short today, so I'm just attaching the http.js and websocket.js revisions as a ZIP file, if you'd like try it out.

wilberforce · 2021-12-28T01:27:11Z

Hi,

I put the your changes here so I could see the differences easily:
https://github.com/Moddable-OpenSource/moddable/compare/public...wilberforce:websocket-http-attach?expand=1

It is tidy with the attach/detach.

Do you propose changing the middleware to use this websocket implementation?

Happy new year...
Rhys

phoddie · 2022-01-05T22:28:41Z

(Apologies for going quiet. My GitHub in-box got messed up, destroying one of my to-do lists...)

I put the your changes here so I could see the differences easily:

Cool, thanks.

It is tidy with the attach/detach.

Agreed. And, I think more generally useful.

Do you propose changing the middleware to use this websocket implementation?

Yes. It would also eliminate the need to duplicate a chunk of code from the WebSocket server.

Have you thought about other names than "middleware"? I think of "middleware" as a category of software, not a specific implementation.

wilberforce · 2022-01-06T01:18:24Z

Have you thought about other names than "middleware"? I think of "middleware" as a category of software, not a specific implementation.

I was using the naming convensions from Node's express: https://expressjs.com/en/guide/using-middleware.html as it's using similiar chained techiques, although I'm using a class implemention.

wilberforce · 2022-02-01T23:03:00Z

@phoddie

Have you thought about other names than "middleware"?

Following the music theme of pico I thought of using a musical term for middle - the most appropriate seemed to be bridge,

Does that work?

BridgeWebsocket, BridgeHotspot

wilberforce · 2022-02-02T20:14:09Z

@phoddie
What do you think of calling the folder bridge instead of middleware?

I'm not sure I need the websocket attach method, as I could follow the pattern of the client implemention and pass the socket to the constructor of the Server - then use the same init code.

phoddie · 2022-02-02T22:18:51Z

I do prefer bridge to middleware. It is reasonably descriptive here.

wilberforce · 2022-02-03T04:32:43Z

@phoddie

I've been putting an example together - pretty happy with it.... I'll move it to examples and post tomorrow, and add a readme for the build steps.... By the way is there are a way of getting mcconfig -d -m to call a script fiest ( I would call the site.zip build step)

phoddie · 2022-02-03T04:38:16Z

By the way is there are a way of getting mcconfig -d -m to call a script first ( I would call the site.zip build step)

There's no way to run a script directly. Recipes is a way to modify the build for certain files, but it may not be what you are looking for here.

wilberforce · 2022-02-03T22:23:38Z

@phoddie
I've just pushed the example code and revised bridge middleware. This contains the detach for HttpDerver and the socket in the constructor of the WebSocket

wilberforce · 2022-02-05T21:07:26Z

There's no way to run a script directly. Recipes is a way to modify the build for certain files, but it may not be what you are looking for here.

I've done an npm script that builds the zip and then runs mcconfig

phoddie · 2022-02-10T00:39:08Z

I've done an npm script that builds the zip and then runs mcconfig

That's probably the best solution at the moment. I'd like to have the manifest provide a way to create a ZIP file from sources, but that's a bit more work that I've time for right now.

Apologies for not getting back to this sooner. I've been distracted elsewhere. I'd like to see this land, so I'll find some time to review.

wilberforce · 2022-02-10T02:13:39Z

I'd like to have the manifest provide a way to create a ZIP file from sources, but that's a bit more work that I've time for right now.
This build uses a cool bit ok kit called wmr which is self contained - only about 3Mb and does all the cunning on the fly reloads and packages the builds, which includes zipping.

Apologies for not getting back to this sooner. I've been distracted elsewhere. I'd like to see this land, so I'll find some time to review.
Cool - a few things to discuss.
One thing I'm struggling with is the preload of bridge/websocket, when I add to the manifest:

mcconfig -d -m
# xsc main.xsb
# xsl modules
# exception: (host): import Server not found!
### SyntaxError: (host): import Server not found!

It can't find websocket/Server and is related to having websocket/websocket and bridge/websocket

phoddie · 2022-02-14T21:41:07Z

This is great example that weaves together many different network tools and techniques frequently used in projects that want to support a web service on the local network. I'm not sure about where it below. A couple observations:

The examples directory in the Moddable SDK is intended to be relatively small projects that show how to use one or two features.
There are many different ways to build an extensible HTTP server in JavaScript in the Moddable SDK. In addition to your httpbridge, we have a couple of example of an Express-style implementations, and a recent Fastify style one.

I suggest we start by putting this in $MODDABLE/contributed/httpbridge/ with example and modules subdirectories. That will generate experience with more people using and evolve from there.

One detail about the WebSockets patch. It invokes callbacks directly from within the attach function which the Moddable SDK generally tries to avoid as it can be unexpected. This version avoid thats at the expense of a bit a code duplication. It also generates a socket readable message so that any already received data is processed immediately.

import Timer from "timer";
//...

export class Client {
//...
	close() {
		this.socket?.close();
		delete this.socket;
		
		if (this.timer)
			Timer.clear(this.timer);
		delete this.timer;
	}
}
//...

export class Server {
	#listener;
	constructor(dictionary = {}) {
		if (null === dictionary.port)
			return;

		this.#listener = new Listener({port: dictionary.port ?? 80});
		this.#listener.callback = () => {
			const socket = new Socket({listener: this.#listener});
			const request = new Client({socket});
			request.doMask = false;
			socket.callback = server.bind(request);
			request.state = 1;		// already connected socket
			request.callback = this.callback;		// transfer server.callback to request.callback
			request.callback(Server.connect, this);	// tell app we have a new connection
		};
	}
	close() {
		this.#listener?.close();
		this.#listener = undefined;
	}
	attach(socket) {
		const request = new Client({socket});
		request.doMask = false;
		socket.callback = server.bind(request);
		request.state = 1;		// already connected socket
		request.callback = this.callback;		// transfer server.callback to request.callback
		request.state = 2;
		request.flags = 0;

		request.timer = Timer.set(() => {
			delete request.timer;
			request.callback(Server.connect, this);	// tell app we have a new connection
			socket.callback(2, socket.read());
		});
	}
};

phoddie · 2022-03-08T21:23:37Z

Apologies for the delay. I've done some work on the WebSockets attach implementation. That will be in the next Moddable SDK update later this week. It includes an example and updated docs. Once that lands, please confirm that your contribution still works and then we can merge this PR!

Notes in brief:

a. I believe it's cleaner to pass the socket to the Server like the Client code does rather than hyjacking a null port

It isn't hijacking the port, but using a null for the port to signal that no listener needs to be created.

b. It's adding a dependancy to Timer that was not there before

It is. But it is unavoidable in order to maintain the JavaScript convention that callbacks generally should not be invoked from inside a user function call.

c. Code is duplicated

Resolved.

wilberforce · 2022-03-08T22:07:05Z

@phoddie
I could not get your example code you posted earlier to work with the timer callback. The this instance on the callback was incorrect.

I'll take a look at your code when it's posted.

phoddie · 2022-03-08T22:37:42Z

I could not get your example code you posted earlier to work with the timer callback. The this instance on the callback was incorrect.

Curious. FWIW – I just did a quick check and verified that each server request is called with a unique this and that the value is consist for callbacks to the request. If you see otherwise with the update, just let me know.

wilberforce · 2022-03-08T22:45:56Z

In my pull request I did the callback in the constructor - it was pretty clean.

Are you able to attach or point me to your websocket.js so I can try it?

phoddie · 2022-03-08T23:00:00Z

In my pull request I did the callback in the constructor - it was pretty clean.

Clean or no, we can't do that: it breaks the rules.

phoddie · 2022-03-08T23:31:08Z

Are you able to attach or point me to your websocket.js so I can try it?

Here you go. Contains the latest websockets.js and a simple test app

wilberforce · 2022-03-09T01:50:47Z

I changed to use your implementation.

The issue I have starts here:

stepping:

So there is no more data in the socket so the handshake does not finish.

This is the code from the bridge websocket:

handler(req, message, value, etc) {
		switch (message) {
			case HTTPServer.status: 
				if ( value === this.#path ) {
					WebSocketUpgrade.bridge=this;
					const socket = this.parent.detach(req);
					const websocket = new WebSocketUpgrade({port:null});
					websocket.attach(socket);
					return;
				}
				break;
		}
		return this.next?.handler(req, message, value, etc);
	}

wilberforce · 2022-03-09T02:13:11Z

If I add debugging to your websocket code:

	attach(socket) {
		const request = addClient(socket, 2, this.callback);
		let bytesAvailable = socket.read()
		trace( `bytesAvailable attach: ${bytesAvailable}\n`);
		request.timer = Timer.set(() => {
			delete request.timer;
			request.callback(Server.connect, this);	// tell app we have a new connection
			let bytesAvailable = socket.read();
			trace( `bytesAvailable timer: ${bytesAvailable}\n`);
			socket.callback(2, socket.read());
		});
	}

C:\Users\rhys\Projects\moddable\modules\network\websocket\websocket.js (312) # Break: breakpoint!
bytesAvailable attach: 492
C:\Users\rhys\Projects\moddable\modules\network\websocket\websocket.js (312) # Break: breakpoint!
C:\Users\rhys\Projects\moddable\modules\network\websocket\websocket.js (312) # Break: breakpoint!
bytesAvailable timer: 0

So something is consuming the buffer.. ?

wilberforce · 2022-03-09T02:56:17Z

If I make addClient a member function in your websocket.js, this works for my if I bypass attach and do the same thing - without the timer:

if ( value === this.#path ) {
	WebSocketUpgrade.bridge=this;
	const socket = this.parent.detach(req);
	const websocket = new WebSocketUpgrade({port:null});
	//websocket.attach(socket);
	const request = websocket.addClient(socket, 2, this.callback);
	let bytesAvailable = socket.read();
	trace( `bytesAvailable attach: ${bytesAvailable}\n`);
	request.callback(WebSocketUpgrade.connect, this);	// tell app we have a new connection
	socket.callback(2, socket.read());
	return;
}

wilberforce · 2022-03-09T02:58:34Z

I've gone back to your example. In the console I only see ws connect and the handshake does not complete.

If I take the timeout out it , and do the callback and socket read it works.

What is the reason for the callback?

phoddie · 2022-03-09T04:12:18Z

Let's try to get on the same page. Here's what I just did:

Grabbed a clean copy of the Moddable SDK from GitHub
Added the websockets.js and httpserverwithwebsockets example from the ZIP above
Built Moddable SDK tools
cd $MODDABLE/examples/network/http/httpserverwithwebsockets
mcconfig -d -m -p sim
Launch Chrome. Connect to localhost in 3 tabs. Then close the tabs.
mcconfig -d -m -p esp ssid=XX password=YY
In Chrome, connect to 10.0.0.ZZ in 3 tabs. Then close the tabs.
mcconfig -d -m -p esp32 ssid=XX password=YY
In Chrome, connect to 10.0.0.ZZ in 3 tabs. Then close the tabs.

The output in xsbug looks something like this (taken from the ESP32 run):

Wi-Fi connected to "My Home Wi-Fi"
IP address 10.0.0.25
ws connect
ws handshake
ws message received: {"hello":"world"}
ws connect
ws handshake
ws message received: {"hello":"world"}
ws connect
ws handshake
ws message received: {"hello":"world"}
ws close
ws connect
ws handshake
ws message received: {"hello":"world"}
ws close
ws close
ws close

I believe you are using Windows, so that's different. But, the ESP8266 and ESP32 builds should give the same result.

wilberforce · 2022-03-09T20:07:58Z

I'm not not winning here.
a. the latest build does not include your new websocket or example, as it's not been committed yet.
b. I don't have the esp build tools setup.
c. Building for esp32 target I get:

[4/79] Building C object esp-idf/libsodium/CMakeFiles/__idf_libso...cryptsalsa208sha256/nosse/pwhash_scryptsalsa208sha256_nosse.c.objFAILED: esp-idf/libsodium/CMakeFiles/__idf_libsodium.dir/f0523bb28a90ed863a6966e11702d0d5/scryptsalsa208sha256/nosse/pwhash_scryptsalsa208sha256_nosse.c.obj
ccache C:\Users\rhys\.espressif\tools\xtensa-esp32-elf\esp-2021r2-patch2-8.4.0\xtensa-esp32-elf\bin\xtensa-esp32-elf-gcc.exe -DCONFIGURED -DHAVE_WEAK_SYMBOLS -DMBEDTLS_CONFIG_FILE=\"mbedtls/esp_config.h\" -DNATIVE_LITTLE_ENDIAN -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -Iconfig -IC:/Users/rhys/esp32/esp-idf/components/libsodium/libsodium/src/libsodium/include -IC:/Users/rhys/esp32/esp-idf/components/libsodium/port_include -IC:/Users/rhys/esp32/esp-idf/components/libsodium/libsodium/src/libsodium/include/sodium -IC:/Users/rhys/esp32/esp-idf/components/libsodium/port_include/sodium -IC:/Users/rhys/esp32/esp-idf/components/libsodium/port -IC:/Users/rhys/esp32/esp-idf/components/newlib/platform_include -IC:/Users/rhys/esp32/esp-idf/components/freertos/include -IC:/Users/rhys/esp32/esp-idf/components/freertos/include/esp_additions/freertos -IC:/Users/rhys/esp32/esp-idf/components/freertos/port/xtensa/include -IC:/Users/rhys/esp32/esp-idf/components/freertos/include/esp_additions -IC:/Users/rhys/esp32/esp-idf/components/esp_hw_support/include -IC:/Users/rhys/esp32/esp-idf/components/esp_hw_support/include/soc -IC:/Users/rhys/esp32/esp-idf/components/esp_hw_support/include/soc/esp32 -IC:/Users/rhys/esp32/esp-idf/components/esp_hw_support/port/esp32/. -IC:/Users/rhys/esp32/esp-idf/components/heap/include -IC:/Users/rhys/esp32/esp-idf/components/log/include -IC:/Users/rhys/esp32/esp-idf/components/lwip/include/apps -IC:/Users/rhys/esp32/esp-idf/components/lwip/include/apps/sntp -IC:/Users/rhys/esp32/esp-idf/components/lwip/lwip/src/include -IC:/Users/rhys/esp32/esp-idf/components/lwip/port/esp32/include -IC:/Users/rhys/esp32/esp-idf/components/lwip/port/esp32/include/arch -IC:/Users/rhys/esp32/esp-idf/components/soc/include -IC:/Users/rhys/esp32/esp-idf/components/soc/esp32/. -IC:/Users/rhys/esp32/esp-idf/components/soc/esp32/include -IC:/Users/rhys/esp32/esp-idf/components/hal/esp32/include -IC:/Users/rhys/esp32/esp-idf/components/hal/include -IC:/Users/rhys/esp32/esp-idf/components/hal/platform_port/include -IC:/Users/rhys/esp32/esp-idf/components/esp_rom/include -IC:/Users/rhys/esp32/esp-idf/components/esp_rom/include/esp32 -IC:/Users/rhys/esp32/esp-idf/components/esp_rom/esp32 -IC:/Users/rhys/esp32/esp-idf/components/esp_common/include -IC:/Users/rhys/esp32/esp-idf/components/esp_system/include -IC:/Users/rhys/esp32/esp-idf/components/esp_system/port/soc -IC:/Users/rhys/esp32/esp-idf/components/esp_system/port/public_compat -IC:/Users/rhys/esp32/esp-idf/components/esp32/include -IC:/Users/rhys/esp32/esp-idf/components/xtensa/include -IC:/Users/rhys/esp32/esp-idf/components/xtensa/esp32/include -IC:/Users/rhys/esp32/esp-idf/components/driver/include -IC:/Users/rhys/esp32/esp-idf/components/driver/esp32/include -IC:/Users/rhys/esp32/esp-idf/components/esp_pm/include -IC:/Users/rhys/esp32/esp-idf/components/esp_ringbuf/include -IC:/Users/rhys/esp32/esp-idf/components/efuse/include -IC:/Users/rhys/esp32/esp-idf/components/efuse/esp32/include -IC:/Users/rhys/esp32/esp-idf/components/vfs/include -IC:/Users/rhys/esp32/esp-idf/components/esp_wifi/include -IC:/Users/rhys/esp32/esp-idf/components/esp_event/include -IC:/Users/rhys/esp32/esp-idf/components/esp_netif/include -IC:/Users/rhys/esp32/esp-idf/components/esp_eth/include -IC:/Users/rhys/esp32/esp-idf/components/tcpip_adapter/include -IC:/Users/rhys/esp32/esp-idf/components/esp_phy/include -IC:/Users/rhys/esp32/esp-idf/components/esp_phy/esp32/include -IC:/Users/rhys/esp32/esp-idf/components/esp_ipc/include -IC:/Users/rhys/esp32/esp-idf/components/app_trace/include -IC:/Users/rhys/esp32/esp-idf/components/esp_timer/include -IC:/Users/rhys/esp32/esp-idf/components/mbedtls/port/include -IC:/Users/rhys/esp32/esp-idf/components/mbedtls/mbedtls/include -IC:/Users/rhys/esp32/esp-idf/components/mbedtls/esp_crt_bundle/include -mlongcalls -Wno-frame-address -ffunction-sections -fdata-sections -Wall -Werror=all -Wno-error=unused-function -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wextra -Wno-unused-parameter -Wno-sign-compare -ggdb -Os -freorder-blocks -fmacro-prefix-map=C:/Users/rhys/Projects/moddable/build/tmp/esp32/m5atom_lite/debug/httpserverwithwebsockets/xsProj-esp32=. -fmacro-prefix-map=C:/Users/rhys/esp32/esp-idf=IDF -fstrict-volatile-bitfields -Wno-error=unused-but-set-variable -fno-jump-tables -fno-tree-switch-conversion -std=gnu99 -Wno-old-style-declaration -D_GNU_SOURCE -DIDF_VER=\"v4.4-dirty\" -DESP_PLATFORM -D_POSIX_READER_WRITER_LOCKS -Wno-unused-function -MD -MT esp-idf/libsodium/CMakeFiles/__idf_libsodium.dir/f0523bb28a90ed863a6966e11702d0d5/scryptsalsa208sha256/nosse/pwhash_scryptsalsa208sha256_nosse.c.obj -MF esp-idf\libsodium\CMakeFiles\__idf_libsodium.dir\f0523bb28a90ed863a6966e11702d0d5\scryptsalsa208sha256\nosse\pwhash_scryptsalsa208sha256_nosse.c.obj.d -o esp-idf/libsodium/CMakeFiles/__idf_libsodium.dir/f0523bb28a90ed863a6966e11702d0d5/scryptsalsa208sha256/nosse/pwhash_scryptsalsa208sha256_nosse.c.obj -c C:/Users/rhys/esp32/esp-idf/components/libsodium/libsodium/src/libsodium/crypto_pwhash/scryptsalsa208sha256/nosse/pwhash_scryptsalsa208sha256_nosse.c
C:/Users/rhys/esp32/esp-idf/components/libsodium/libsodium/src/libsodium/crypto_pwhash/scryptsalsa208sha256/nosse/pwhash_scryptsalsa208sha256_nosse.c:383:1: fatal error: opening dependency file esp-idf\libsodium\CMakeFiles\__idf_libsodium.dir\f0523bb28a90ed863a6966e11702d0d5\scryptsalsa208sha256\nosse\pwhash_scryptsalsa208sha256_nosse.c.obj.d: No such file or directory

d. I'm still getting the same issue with the windows sim. I've echoed IP and have the socket available bytes:

IP Address: 172.20.224.1
bytesAvailable attach: 492
ws connect
bytesAvailable timer: 0

I don't understand why the timer is required - why can't the callback and socket be read straight away?

Thanks

phoddie · 2022-03-09T20:18:23Z

Thank you for your patience.

a. the latest build does not include your new websocket or example, as it's not been committed yet.

That's correct. That is why step 2 above is "Added the websockets.js and httpserverwithwebsockets example from the ZIP above."

c. Building for esp32 target I get...

That appears unrelated to this topic. The Moddable SDK did recently update to ESP-IDF 4.4. Perhaps you don't have the expected version. In any case, this isn't the issue to solve that.

I don't understand why the timer is required - why can't the callback and socket be read straight away?

A synchronous function, such as attach, should not cause any callbacks to be invoked. That is the general model that the networking protocols follow as does Ecma-419:

A callback function may only be invoked when no script is running in its host virtual machine to respect the single-thread evaluation semantics of ECMAScript. This means that callbacks may not be invoked by the instance from within its public method calls, including the constructor.

This is important because callbacks from within a synchronous function are often unexpected and lead to difficult to track down bugs.

It may be that the issue you are seeing is isolated to Windows, perhaps due to a difference in the socket implementation there. Before going down that road further, I think it is important that we are able to reproduce any of the same results. It seems that ESP32 is the place for that.

wilberforce · 2022-03-09T21:25:25Z

I have installed a clean idf 4.4. Helloword builds correctly.

Still getting this error:
C:/Users/rhys/esp32/esp-idf/components/libsodium/libsodium/src/libsodium/crypto_pwhash/scryptsalsa208sha256/nosse/pwhash_scryptsalsa208sha256_nosse.c:383:1: fatal error: opening dependency file esp-idf\libsodium\CMakeFiles\__idf_libsodium.dir\f0523bb28a90ed863a6966e11702d0d5\scryptsalsa208sha256\nosse\pwhash_scryptsalsa208sha256_nosse.c.obj.d: No such file or directory }

Appears to be related to this:

https://esp32.com/viewtopic.php?t=14651

the regedit hack was already set on my machine.

I copied the project form:
C:\Users\rhys\Projects\moddable\modules\network\websocket\httpserverwithwebsockets
to:
C:\Users\rhys\source\repos\hws

and now its builds. Grrrr

The esp32 build works for me:

Wi-Fi connected to "ssid"
IP address 192.168.15.137
IP Address: 192.168.15.137
bytesAvailable attach: 498
ws connect
bytesAvailable timer: 498
ws handshake
ws message received: {"hello":"world"}

So it appears to be related to the windows socket implementation in the simulator.

andycarle · 2022-03-09T21:49:23Z

I can't tell you how much I hate that path length issue.

A way to avoid it is to turn off ccache, but that's a pretty heavy-handed fix. I should probably update the Windows instructions to suggest cloning at c:\moddable. But even there you sometimes run into problems with long application names.

wilberforce · 2022-03-09T22:18:10Z

I can't tell you how much I hate that path length issue.

A way to avoid it is to turn off ccache, but that's a pretty heavy-handed fix. I should probably update the Windows instructions to suggest cloning at c:\moddable. But even there you sometimes run into problems with long application names.

Thanks @andycarle
In this case it seems related to the actual project path, this fails: C:\Users\rhys\Projects\moddable\modules\network\websocket\httpserverwithwebsockets

and this builds:
C:\Users\rhys\Projects\moddable\modules\network\websocket\httpws

Any idea why socket is behaving differently in the windows simulator?

phoddie · 2022-03-10T02:50:44Z

Windows socket should now behave like macOS, ESP32, and ESP8266. Thanks to @andycarle for the debugging help.

wilberforce · 2022-03-10T03:23:23Z

Windows socket should now behave like macOS, ESP32, and ESP8266. Thanks to @andycarle for the debugging help.

Thanks guys. I can confirm the fix is working for me! I can't say I understand the changes - but it works!

Fantastic support.

wilberforce · 2022-03-10T04:02:06Z

Upload.from.GitHub.for.iOS.MOV

Quick video showing web sockets updating multiple devices

phoddie · 2022-03-10T04:14:14Z

It works!!

phoddie · 2022-03-24T00:26:19Z

Somehow this hasn't been merged yet. I think it is ready to go? @wilberforce - good to go?

wilberforce · 2022-03-24T00:30:16Z

@phoddie
Yes - good to go

Add http middleware and websocket upgrade on existing listener

7b60140

add example, rename middleware to bridge

a5b1e38

wilberforce added 2 commits February 8, 2022 16:25

fix Server websocket error - however can't preload bridge/websocket

283b228

Add App with methods mapping ws actions

e0e909d

wilberforce added 2 commits February 10, 2022 14:47

Add save/restore model to preferences

3081725

remove unused npm modules

7fec23a

wilberforce added 2 commits February 10, 2022 15:16

move #model to app

4c4cd85

fix npm dependancies

9ed878c

change to @phoddies websocket

6bec2ad

mkellner pushed a commit that referenced this pull request Mar 9, 2022

WebSocket server creates new connection from existing socket #755

72c9f34

update ws

939ba3d

mkellner pushed a commit that referenced this pull request Mar 10, 2022

Windows socket fixes (see #755) (code from @phoddie)

5df3b55

Merge branch 'Moddable-OpenSource:public' into http-server-middleware

1a56d3f

mkellner pushed a commit that referenced this pull request Mar 25, 2022

Add http middleware and websocket upgrade on existing listener #755

5954fc3

mkellner pushed a commit that referenced this pull request Aug 11, 2023

WebSocket server creates new connection from existing socket #755

01c666e

mkellner pushed a commit that referenced this pull request Aug 11, 2023

Windows socket fixes (see #755) (code from @phoddie)

15dd8b8

Add http middleware and websocket upgrade on existing listener #755

Are you sure you want to change the base?

Add http middleware and websocket upgrade on existing listener #755

Conversation

wilberforce commented Dec 21, 2021

phoddie commented Dec 22, 2021

wilberforce commented Dec 22, 2021

wilberforce commented Dec 22, 2021 • edited

phoddie commented Dec 23, 2021

phoddie commented Dec 23, 2021

wilberforce commented Dec 24, 2021 • edited

phoddie commented Dec 24, 2021

wilberforce commented Dec 25, 2021

phoddie commented Dec 26, 2021

wilberforce commented Dec 28, 2021

phoddie commented Jan 5, 2022

wilberforce commented Jan 6, 2022

wilberforce commented Feb 1, 2022 • edited

wilberforce commented Feb 2, 2022

phoddie commented Feb 2, 2022

wilberforce commented Feb 3, 2022

phoddie commented Feb 3, 2022

wilberforce commented Feb 3, 2022 • edited

wilberforce commented Feb 5, 2022

phoddie commented Feb 10, 2022

wilberforce commented Feb 10, 2022

phoddie commented Feb 14, 2022

phoddie commented Mar 8, 2022

wilberforce commented Mar 8, 2022

phoddie commented Mar 8, 2022

wilberforce commented Mar 8, 2022

phoddie commented Mar 8, 2022

phoddie commented Mar 8, 2022 • edited

wilberforce commented Mar 9, 2022

wilberforce commented Mar 9, 2022

wilberforce commented Mar 9, 2022

wilberforce commented Mar 9, 2022 • edited

phoddie commented Mar 9, 2022

wilberforce commented Mar 9, 2022

phoddie commented Mar 9, 2022

wilberforce commented Mar 9, 2022

andycarle commented Mar 9, 2022

wilberforce commented Mar 9, 2022 • edited

phoddie commented Mar 10, 2022

wilberforce commented Mar 10, 2022 • edited

wilberforce commented Mar 10, 2022 • edited

phoddie commented Mar 10, 2022

phoddie commented Mar 24, 2022

wilberforce commented Mar 24, 2022

wilberforce commented Dec 22, 2021 •

edited

wilberforce commented Dec 24, 2021 •

edited

wilberforce commented Feb 1, 2022 •

edited

wilberforce commented Feb 3, 2022 •

edited

phoddie commented Mar 8, 2022 •

edited

wilberforce commented Mar 9, 2022 •

edited

wilberforce commented Mar 9, 2022 •

edited

wilberforce commented Mar 10, 2022 •

edited

wilberforce commented Mar 10, 2022 •

edited