PHP: Support http:// https:// and ssl:// stream wrappers #1093

adamziel · 2024-03-07T12:09:23Z

What is this PR doing?

Intercepts all network traffic coming from PHP and handles http:// and https:// requests using fetch(). This enables using the native networking features of PHP without implementing custom transport classes.

🚧 Work in progress – this PR needs discussion and cleaning up before it can be shipped 🚧

How is it implemented?

Emscripten can be configured to stream all network traffic through a WebSocket. @php-wasm/node and wp-now use that to access the internet via a local WebSocket->TCP proxy, but the in-browser version of WordPress Playground exposes no such proxy.

This PR ships a "fake" WebSocket implementation that doesn't initiate any ws:// connection. Instead, it mocks the WebSocket interface and analyzes the connection address and transmitted bytes to infer a corresponding fetch() call.

In case of HTTP, it parses the request text, extracts the method, path, and headers (body TBD), and feeds that information to a fetch() call. Then, as the response status, headers, and the data stream comes in, it rewrites it as raw bytes and pretends to emit them as incoming WebSocket data.

In case of HTTPS, it uses the node-forge package to start a HTTPS server with a self-signed certificate that is also added to PHP CA store via the openssl.cafile PHP.ini setting. The outbound traffic is piped to node-forge which handles the SSL handshake, the encryption, and yields unencrypted HTTP request bytes. They are treated the same as in the previous paragraph, and then the response is piped back to node-forge for encryption and then, finally, piped back to PHP.

The current implementation is naive and optimistically assumes the data is intended for either HTTP or HTTPS use.

Upsides

file_get_contents(), fopen(), fsockopen() and other functions using stream wrappers will now work with http://, https://, and ssl:// URLs!
libcurl can be now be supported without any customizations as the network traffic handler doesn't care about the library that initiated the connection.

Downsides

node-forge doesn't seem to work well with the latest TLS so we'd have to force-downgrade it in PHP. It doesn't seem like a big deal in the browser because all SSL security is provided by fetch(), but it's something to note. This approach is , of course, unsuitable and unnecessary outside of web browsers.
Raw sockets and CORS-less URLs still can't be supported this way.
I'm not sure whether we can reliably distinguish between HTTP / HTTPS traffic and a byte transmission via fsockopen('ssl://somesite.com'). Perhaps not. In this case we could ship a naive heuristic that would cover the majority of cases, and then patch PHP to provide an explicit flag "I'm about to request something via HTTPS".
node-forge is slow and the certificates are generated synchronously. It would be far better to use the browser-native asynchronous API like crypto.subtle.generateKey().
HTTPS adds noticeable overhead. Sticking to the FetchTransport in WordPress sounds like a good idea since it triggers fetch() directly and without an encryption layer.
I think each domain requires a separate cert, which means a bit of slowness added to the first request to every domain. That sounds like a fair trade-off for providing the much-requested networking support.

Remaining work

Discuss the approach
Clean up the code and the import structure
Remove as much synchronous processing as possible
Generate SSL certificates lazily

Testing Instructions

Run npm install
Go to http://localhost:5400/website-server/#{%22landingPage%22:%22/network-test.php%22,%22preferredVersions%22:{%22php%22:%228.0%22,%22wp%22:%22latest%22},%22phpExtensionBundles%22:[%22kitchen-sink%22],%22steps%22:[{%22step%22:%22writeFile%22,%22path%22:%22/wordpress/network-test.php%22,%22data%22:%22%3C?php%20echo%20'Hello-dolly.zip%20downloaded%20from%20https://downloads.wordpress.org/plugin/hello-dolly.1.7.3.zip%20has%20this%20many%20bytes:%20';%20var_dump(strlen(file_get_contents('https://downloads.wordpress.org/plugin/hello-dolly.1.7.3.zip')));%22}]}
Confirm the page says Hello-dolly.zip downloaded from https://downloads.wordpress.org/plugin/hello-dolly.1.7.3.zip has this many bytes: int(1887)

That Blueprint above runs the following PHP code:

<?php
echo 'Hello-dolly.zip downloaded from https://downloads.wordpress.org/plugin/hello-dolly.1.7.3.zip has this many bytes: ';
var_dump(strlen(file_get_contents('https://downloads.wordpress.org/plugin/hello-dolly.1.7.3.zip')));

Related to #724

cc @dmsnell @bgrgicak @brandonpayton @ThomasTheDane

bgrgicak

It's great to see progress on this 🚀

Should this eventually become a separate package?

A naive question. Is HTTPS necessary or could we just use HTTP internally?

bgrgicak · 2024-03-08T07:11:09Z

@adamziel When I opened the test link (on Linux) I got back an error Hello-dolly.zip downloaded from https://downloads.wordpress.org/plugin/hello-dolly.1.7.3.zip has this many bytes: int(1887).

adamziel · 2024-03-08T07:38:46Z

That's the expected output and not an error :-)

adamziel · 2024-03-08T07:41:56Z

A naive question. Is HTTPS necessary or could we just use HTTP internally?

How would that work?

bgrgicak · 2024-03-08T08:28:56Z

That's the expected output and not an error :-)

I should read all the testing instructions before I comment 😕

bgrgicak · 2024-03-08T13:48:55Z

How would that work?

I imagined that we could just send these requests without having an SSL server.
WordPress and CURL requests can be configured to ignore SSL certificates.
But that's probably a bad idea.

dmsnell

looks like this is still early on in its development?

were there any specific things you'd like review on?

dmsnell · 2024-03-08T22:21:55Z

packages/php-wasm/web/src/lib/web-php.ts

@@ -44,6 +48,560 @@ const fakeWebsocket = () => {
 	};
 };

+async function generateKeys() {


this and createCertificate() would both be very good functions to document, specifically why they exist, what kind of keys and certificates they generate, and why

dmsnell · 2024-03-08T22:23:53Z

packages/php-wasm/web/src/lib/web-php.ts

+		},
+		{
+			name: 'localityName',
+			value: 'Blacksburg',


we probably don't need all of these attributes do we? if not, maybe we should have a good explanation for why we're using the very specific values we are. we could probably make up a virtual/imaginary Playground land in WordPressWorld so that if someone were to examine the certificates it would be clearer that these are for a virtual system and not supposed to be actual legitimate signed certificates

dmsnell · 2024-03-08T22:25:11Z

packages/php-wasm/web/src/lib/web-php.ts

+			sslCA: true,
+			emailCA: true,
+			objCA: true,
+		},


we surely don't need all these capabilities in the certificate do we? it shouldn't hurt anything, but do we expect, for example, the Playground to sign email signatures?

Good point, most of these should be fine to remove

dmsnell · 2024-03-08T22:26:19Z

packages/php-wasm/web/src/lib/web-php.ts

+							10
+						);
+
+						const ws = this;


copy/paste or AI code?

Quick&dirty prototype, ws is used in a function in line 316 below where this refers to a different object.

adamziel · 2024-03-11T23:34:13Z

were there any specific things you'd like review on?

Mostly the general idea and approach in case there was something off with it.

dmsnell · 2024-03-25T09:34:32Z

Summarizing a conversation @adamziel and I had: this seems like a fine approach, though I think it's a long way away from where it needs to be with naming and documentation.

This is essentially a TLS proxy standing as a middleman between the PHP code and JS code. What it isn't is writing our own security layer, and this is because ultimately all TLS connections on the JS side will be enforced with the security mechanisms that the browser provides.

Proxying through fetch() is challenging because it requires that we grab the data out of the encrypted stream and present it to PHP in a new encrypted stream, acting like it was never translated, thus the middleman.

It looks like we can provide a ReadableStream to fetch as the body of the network request so I think it's even possible to go beyond what's here and provide a generalized socket interface into PHP. I was on my way to testing that out when I ran out of time. There's still the challenge of detecting when a TLS handshake is beginning, but maybe that's not too hard to do.

Ships the Node.js version of PHP built with `--with-libcurl` option to support the curl extension. It also changes two nuances in the overall PHP build process: * It replaces the `select(2)` function using `-Wl,--wrap=select` emcc option instead of patching PHP source code – this enables supporting asynchronous `select(2)` in curl without additional patches. * Brings the `__wrap_select` implementation more in line with `select(2)`, add support for `POLLERR`. * Adds support for polling file descriptors that represent neither child processes nor streams in `poll(2)` – that's because `libcurl` polls `/dev/urandom`. Builds on top of and supersedes #1133 ## Debugging Asyncify problems The [typical way of resolving Asyncify crashes](https://wordpress.github.io/wordpress-playground/architecture/wasm-asyncify/) didn't work during the work on this PR. Functions didn't come up in the error messages and even raw stack traces. The reasons are unclear. [The JSPI build of PHP](#1339) was more helpful as it enabled logging the current stack trace in all the asynchronous calls, which quickly revealed all the missing `ASYNCIFY_ONLY` functions. This is the way to debug any future issues until we fully migrate to JSPI. ## Testing Instructions Confirm the CI checks pass. This PR ships a few new tests specifically targeting networking with curl. ## Related resources * #85 * #1093 --------- Co-authored-by: Adam Zieliński <adam@adamziel.com> Co-authored-by: MHO <yannick@chillpills.io>

…1389) Aligns the boot process between the in-browser Playground Remote and Node-oriented Playground CLI. With this PR, both apps use a similar `createPHP()` function that: * Sets up the SAPI name * Sets up PHP ini entries * Sets up the `/phpinfo.php` route * Sets up platform-level mu0plugins * Proxies filesystem directories from secondary PHP instances to primary * Sets up PHP runtime rotation to avoid OOM errors in long-running primary processes There are still the following discrepancies: * The in-browser PHP sets up a SAPI name conditionally, Node.js one always uses `cli` (it probably shouldn't) * The in-browser PHP uses a custom spawn handler * The in-browser PHP uses a different set of php.ini directives * The in-browser PHP loads more mu-plugins * The Node.js PHP sets up CA certificates for HTTPS connections (the in-browser PHP [will fake the CA chain eventually](#1093)) This is the first step towards a consistent Boot Protocol, see #1379 for more details. ## Testing Instructions * Confirm the CI checks work * Run `bun packages/playground/cli/src/cli.ts server --login`, confirm the server starts without issues, test wp-admin and HTTPS-reliant features like the plugin directories. We'll need a set of unit tests for these new boot-related features, let's create them sooner than later.

adamziel · 2024-05-20T19:19:48Z

node-forge is quite slow when it comes to certificate generation, perhaps https://pkijs.org/docs/examples/certificates-and-revocation/create-and-validate-certificate would be faster

PHP: Support http:// https:// and ssl:// stream wrappers

9ddf54b

adamziel added [Type] Enhancement New feature or request [Feature] PHP.wasm [Aspect] Networking labels Mar 7, 2024

adamziel requested a review from a team as a code owner March 7, 2024 12:09

This was referenced Mar 7, 2024

Blueprints: Explore switching to the PHP implementation #1051

Closed

Error because of lack of cURL and allow_url_fopen #1008

Open

bgrgicak reviewed Mar 8, 2024

View reviewed changes

dmsnell reviewed Mar 8, 2024

View reviewed changes

dmsnell added 2 commits March 16, 2024 11:14

Move TLS logic into a new module.

15007c8

Move another file, decode as latin1

0e8f8db

This was referenced Mar 22, 2024

PHP Class finfo not found #1127

Closed

Network access in the browser #85

Open

Explore curl support #1133

Closed

adamziel added this to the PHP Feature Parity milestone Apr 4, 2024

adamziel mentioned this pull request Apr 29, 2024

Curl extension for the Node.js build of PHP.wasm #1273

Merged

adamziel mentioned this pull request Apr 29, 2024

Support SQLite at the network connection level #1341

Open

adamziel mentioned this pull request May 14, 2024

Playground Boot: Align the boot process between remote.html and CLI #1389

Merged

eherman-godaddy mentioned this pull request May 15, 2024

HTTP POST request to rest endpoint returning Host is unreachable #1403

Open

adamziel mentioned this pull request May 17, 2024

PHP feature parity #763

Open

28 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PHP: Support http:// https:// and ssl:// stream wrappers #1093

PHP: Support http:// https:// and ssl:// stream wrappers #1093

adamziel commented Mar 7, 2024 •

edited

bgrgicak left a comment

bgrgicak commented Mar 8, 2024

adamziel commented Mar 8, 2024

adamziel commented Mar 8, 2024

bgrgicak commented Mar 8, 2024

bgrgicak commented Mar 8, 2024

dmsnell left a comment

dmsnell Mar 8, 2024

dmsnell Mar 8, 2024

dmsnell Mar 8, 2024

adamziel Mar 11, 2024

dmsnell Mar 8, 2024

adamziel Mar 11, 2024

adamziel commented Mar 11, 2024

dmsnell commented Mar 25, 2024

adamziel commented May 20, 2024

PHP: Support http:// https:// and ssl:// stream wrappers #1093

Are you sure you want to change the base?

PHP: Support http:// https:// and ssl:// stream wrappers #1093

Conversation

adamziel commented Mar 7, 2024 • edited

What is this PR doing?

How is it implemented?

Upsides

Downsides

Remaining work

Testing Instructions

bgrgicak left a comment

Choose a reason for hiding this comment

bgrgicak commented Mar 8, 2024

adamziel commented Mar 8, 2024

adamziel commented Mar 8, 2024

bgrgicak commented Mar 8, 2024

bgrgicak commented Mar 8, 2024

dmsnell left a comment

Choose a reason for hiding this comment

dmsnell Mar 8, 2024

Choose a reason for hiding this comment

dmsnell Mar 8, 2024

Choose a reason for hiding this comment

dmsnell Mar 8, 2024

Choose a reason for hiding this comment

adamziel Mar 11, 2024

Choose a reason for hiding this comment

dmsnell Mar 8, 2024

Choose a reason for hiding this comment

adamziel Mar 11, 2024

Choose a reason for hiding this comment

adamziel commented Mar 11, 2024

dmsnell commented Mar 25, 2024

adamziel commented May 20, 2024

adamziel commented Mar 7, 2024 •

edited