Replies: 15 comments 48 replies
-
|
cc @wojtekn @fluiddot @sejas @p-jackson for any wp-now/Studio insights. |
Beta Was this translation helpful? Give feedback.
-
|
Perhaps BrowserFs could solve some of the problems here - it seems to support „overlaying” filesystem layers on top of each other, meaning we could perhaps use a local WordPress directory and add a custom db.php in VFS without affecting the user’s project. Another idea would be to use linux mount:
|
Beta Was this translation helpful? Give feedback.
-
I've looked at Overlay FS before for some actual Systems things, and it seems pretty handle and natural to use. But with Emscripten, how do you imagine we could just "use linux mount"? I'm likely missing a bit of knowledge. |
Beta Was this translation helpful? Give feedback.
-
|
Thinking about this more, I don't think we could just use it. It's all about the system-level support, and our system is a bunch of JavaScript functions in a trench coat :D |
Beta Was this translation helpful? Give feedback.
-
😄 That's a great picture. |
Beta Was this translation helpful? Give feedback.
-
I think that we should allow for two options. Either you give Playground a location and let it add files as needed. Or you give it a populated directory with WordPress. For example Playground web provides a WordPress directory, but only partially. |
Beta Was this translation helpful? Give feedback.
-
|
There's four things that may happen when I mount a directory at document root:
My intention may be to:
I'm not sure whether we should support all combinations of these intentions.
Or, more plainly, here are the combinations I'd support:
Therefore, we'd need the following knobs:
There could be good error messages for all the known failure modes, e.g. trying to mount There's a tie in to Blueprints v2 as the Blueprints library would be responsible for that setup. Let's figure that tactical bit later on once we have a better idea about the high-level requirements. |
Beta Was this translation helpful? Give feedback.
-
We might need explicit "pre-install" and "post-install" mounts. I don't think either option would do the trick on its own:
|
Beta Was this translation helpful? Give feedback.
-
I would copy what Docker does here. Mounting would result in a sync between directories. We could also offer copy as a option for cases where you don't want to modify your local mount point. |
Beta Was this translation helpful? Give feedback.
-
|
This is tempting, but at the same time there are valid reasons to sync only in one direction. Say I have a GitHub monorepo with
I could either:
Here's another scenario: In playground-docs-workflow I don't want any Blueprint-installed plugins to show up in my local directory, but I do want HTML pages created in Playground to be created in my local Perhaps we need two types of mounts:
Technically, a one-way mount could be an "Overlay Filesystem" that reads from MEMFS first and NODEFS second, but writes only to MEMFS. Emscripten's WASMFS will likely need some more months or quarters to provide that feature so I've explored alternative implementation in #1380 – it's incomplete, needs more work, and potentially too slow for us to use, but there's a slight change it would be suitable. |
Beta Was this translation helpful? Give feedback.
-
|
A good stopgap solution until we have overlay fs could be:
|
Beta Was this translation helpful? Give feedback.
-
The more I think of it, this should be enough. Make copy as a special use case and do readonly mounts bu default. Playground on it's own is stateless so keeping the sync one directional makes sense. |
Beta Was this translation helpful? Give feedback.
-
I like this. Real files are projected into Playground space, and only chosen things are synced. This is a safe default and feels like intuitive behavior to me. There might be two ways to perform sync:
|
Beta Was this translation helpful? Give feedback.
-
This should be handled by the developer the same way wp-now does it now. |
Beta Was this translation helpful? Give feedback.
-
|
One of the related issues WordPress/playground-tools#244 |
Beta Was this translation helpful? Give feedback.
-
What if this was internal to Playground? Whenever I mount a folder, Playground checks the existing structure and mounts subfolders. |
Beta Was this translation helpful? Give feedback.
-
My intuition is uncomfortable with this, and I'm going to try to talk it out... With this kind of implicit behavior, it can be easy to use because it automatically reconciles everything and you don't have to say as much. But a natural drawback to this is that the system assumes user intent when that intent is not always clear. How would it work when the user wants to develop both a theme and one of its dependencies in parallel?
I think it is tricky to know how to handle conflicts because users may or may not want to replace pre-existing subdirs. IMO it is better to have users say exactly what they want so there is less doubt. In addition, a user might make not provide what they think they are providing. They can provide an entirely wrong directory or maybe the right directory containing more or less than they expect. Or they might incorrectly guess how the system will implicitly apply their provided directories. The resulting FS may be much different than their mental model suggests, and we have fewer opportunities to catch mistakes because they haven't clearly stated intent. |
Beta Was this translation helpful? Give feedback.
-
This wouldn't work because we can't mount folders inside mounted folders AFAIK. But it could be done by copying files in case you don't need sync. But that would be a hack. |
Beta Was this translation helpful? Give feedback.
-
I agree. The goal of my suggestion was to work around mounting limitations by manually mounting sub-paths to allow multiple mounts on the same existing folder. |
Beta Was this translation helpful? Give feedback.
-
Imported files should override runtime-provided files. This will give developers control. |
Beta Was this translation helpful? Give feedback.
-
|
Imagine we do that and someone exports Playground with a runtime-provided |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
Yeah, WordPress looks for it at the exact location. I’ll see if this can be worked around |
Beta Was this translation helpful? Give feedback.
-
|
It seems like we can avoid creating |
Beta Was this translation helpful? Give feedback.
-
This could be a nice way to ensure we always run the latest version.
We could detect if the SQLite plugin is mounted in wp-content/plugins and not load the internal plugin if it exists. |
Beta Was this translation helpful? Give feedback.
-
I would use /internal whenever possible for Playground dependencies. |
Beta Was this translation helpful? Give feedback.
-
|
Agreed, my current mental model here is "Snapshots are not standalone archives, you need an import tool". That tool would rewrite URLs, install the SQLite integration plugin or import the SQL data etc. |
Beta Was this translation helpful? Give feedback.
-
|
Exporting a "runnable Snapshot" would involve the same tool and produce a zip archive that's hosting-ready. The tool would run during the export process and require all the same details as conversion from the "non-runnable" snapshot to a "runnable" one, so the target site URL, target database etc. Importing such a "runnable snapshot" to Playground would be an inverse transform and would need to know the source site URL, source database etc. However, Thinking about specific transforms:
Playground would be a special case of a "Site" and each runtime would preload its own
This seems to be largely about building the v1 of Site Transfer Protocol for Site->ZIP and ZIP->Site. A larger roadmap here would be something like this:
In Playground, however, for now we can get away with a simplified version:
|
Beta Was this translation helpful? Give feedback.
-
|
About rewriting URLs, here's one of the related issues: WordPress/playground-tools#14 |
Beta Was this translation helpful? Give feedback.
-
I would always include the entire directory. This is what Playground needs to run and it gives users control over what version to use.
I wouldn't include internals. These files are part of Playground and may depend on the current version of Playground. |
Beta Was this translation helpful? Give feedback.
-
A snapshot without WordPress core files can be kilobytes large, but one with WordPress core won't be smaller than a few megs. Also, when exporting files to GitHub or OPFS, there's very few scenarios where you want to include WordPress core.
👍 I think this is the same larger general question as "Are exports ZIPs standalone and ready to run after extracting," and I now think the answer is "no". |
Beta Was this translation helpful? Give feedback.
-
|
If the answer is no, I agree, we shouldn't include WordPress. If we supported mounting like described here the export format could match the folder structure of the entire site, but exclude WordPress. |
Beta Was this translation helpful? Give feedback.
-
|
I see two different parts to this discussion. One is how Core Playground works, this is the unopinionated system that allows us to build modules like Playground Web and Server.
It always expects a fully populated VFS. If a file is overridden or excluded. Playground might break, it won't try to resolve the issue internally.
They accept user input like mount points. They may include some complex logic to stitch the VFS together, pull data from various sources, and handle conflicts. |
Beta Was this translation helpful? Give feedback.
-
|
I agree! I'd like to expose sensible pipelines/primitives from core for the modules to reuse. Many of them have similar requirements and could perhaps just pass their mounts, a few options, and a Blueprint to a |
Beta Was this translation helpful? Give feedback.
-
|
Having a common flexible |
Beta Was this translation helpful? Give feedback.
-
Full snapshotsA full snapshot freezing the entire site in time would include a top-level Partial snapshotsA minimal snapshot relying on the runtime and network resources for setup could ship just a single mu-plugin or a CSS file. However, there's no reliable way of handling a zip archive with just a single file inside. We need a path mapping. The path mapping could be implicit, meaning the zip archive would have a Consider snapshot A with a Or consider snapshot B with a Supporting partial snapshots requires shipping additional metadata to inform the import process. Partial snapshot metadataI'm not yet sure what that metadata should be, but I'm thinking about supporting:
Perhaps that's too much for v1, though, and supporting just |
Beta Was this translation helpful? Give feedback.
-
|
This seems to overlap exactly with the zip-based version of Site Transfer Protocol. |
Beta Was this translation helpful? Give feedback.
-
|
Noodling on the discussion, I came up with this boot flow that we could implement without any new platform-level features – I'd love to learn what you think. Is there any use-case this wouldn't handle? My impression is that this would work pretty well for all the discussed use-cases. I'd still like to map them out separately to be extra sure, e.g. importing from GitHub, mounting OPFS in the browser, multiple local mounts in Node.js, pre-mounting This flow would still affect Conflicting or outdated
|
Beta Was this translation helpful? Give feedback.
-
I think you're right. We don't want partial URL replacement. |
Beta Was this translation helpful? Give feedback.
-
What if we don't rewrite them in the DB, but fix the URL during the redirect? |
Beta Was this translation helpful? Give feedback.
-
We can't do that if it's a different domain because a) we don't have a server running on that other domain, and b) even if we had one, it wouldn't know where to redirect. |
Beta Was this translation helpful? Give feedback.
-
Shouldn't we be able to detect the URL in I took a quick look and link click are using |
Beta Was this translation helpful? Give feedback.
-
|
The service worker is only notified about same-domain navigations |
Beta Was this translation helpful? Give feedback.
-
|
This is a response to the original post. I haven't reviewed the latest comments yet. Some overall thoughts: It would be good favor an explicit language of configuration over magic that makes specific scenarios easy but might not dovetail well with all uses of the system. Maybe it would make sense to have some kind of "setup compiler" that takes explicit declarations about FS and DB and yields a usable WordPress setup (comprised of FS, DB, etc). There would also be a validation step that confirms the result looks like a complete WordPress setup. Something like Compile -> Validate -> Boot. Instead of having questions about how different FS combinations should work, why don't we turn those questions into explicit config options for FS construction? The same could go for questions about DB construction. The main idea is: Compilation of setupAn FS config would be comprised of an arbitrary number of declarations about the filesystem, including identifying specific paths, whether and how to populate files for those paths, and whether each path should be read-only or writable. The result would be a single directory tree. A DB config would be comprised of a declaration of type. The type could be SQLite, MySQL, or custom, where "custom" means the user will rely on what is set up in the FS. The "custom" option could even be used when providing user-provided versions of things like sqlite-database-integration. This is kind of like a platform blueprint where today's blueprints are more like "runtime" blueprints in comparison. As @bgrgicak mentioned
Making setup explicit doesn't mean that the UX for Playground needs to overwhelm the user with options. There could be sensible defaults and thoughtful config options, but with explicit setup compilation, a platform language would be there for deeper cases, less common cases, and unanticipated cases. |
Beta Was this translation helpful? Give feedback.
-
I like that! Note this would be a runtime data structure rather than a static declaration, since things like OPFS mounts are only exposed via user interaction. An in-runtime configuration for setting up a runtime sounds fine. We could lean on TypeScript type system for validation.
I like that! Each entry is characterized by:
Some combinations would lead to errors, e.g. zip + two-way, or nested mounts.
+1
It is similar, isn't it? I still wonder where to draw the line. Since we'll migrate to a PHP Blueprints implementation, I'd say anything that can't be handled by that library shouldn't be a part of Blueprints – e.g. mounts. |
Beta Was this translation helpful? Give feedback.
-
Yep. And it seems these constraints could even be checked by the type system.
Agreed. They should be separate things. A foundation cannot be defined in the terms of everything that rests upon it. 😄 This discussion is making me philosophical... Like Godel's proof which is roughly:
|
Beta Was this translation helpful? Give feedback.
-
|
This reminds me of Gödel, Escher, Bach – what a great book! |
Beta Was this translation helpful? Give feedback.
-
With explicit declaration, this question would be answered by the user.
Perhaps this could be part an explicit declaration as well, with more format types supported over time. Validate that FS and DB config matches expected WP format. If validation fails, we might continue loading Playground but validation errors could be listed in the event a crash or failure to load WordPress.
Users should be explicit about what they want here. But could we have sensible defaults that minimize friction, surprises, and the need juggle a bunch of things in one's head? Are there ways we could store different WP versions and builds by key so they don't conflict, and we can mount the one desired?
All decisions should be made as part of initial config. If we are talking about default Playground UI experience on the web, we need to design so that no surprise conflicts are possible. If we are letting a user mutate WP state, we need to make sure conflicts are impossible. QUESTION: What kinds of conflicts are possible today? NOTE: I don't yet know how playground.internal is used but will find out.
Maybe /internal is a good place? Or /data?
This needs to be configured explicitly. And perhaps the default is that it is always installed unless the configuration says otherwise.
IMO it is clearer and cleaner to keep platform mu-plugins separate from user mu-plugins.
This should be declared explicitly.
Can we detect when running the installer is needed? This should be checked as part of pre-boot validation. Perhaps users should say whether they expect WP tables to exist already or whether they should be auto-created by Playground if missing.
I may not understand the context, but naively... Could we mount as read-only or copy the files into temporary storage?
Read-only sources should be the default. Writability should be explicitly configured as part of setup.
Conceptually, I don't think this is a Playground use case. Or if it is, it should be related with according to Site Transfer Protocol.
This sounds the same as (or similar to) a "WordPress setup" compiler.
Everything necessary to run the configuration in Playground, even if some elements are merely references to external resources.
What kinds of changes?
It feels like we might want two kinds of exports:
The casual export should be lighter and work most of the time, but if someone wants an exact copy of the setup, it would be good to be able to have something like an "archival" export. It would be helpful for stability and bug repro to export exact software for a given Playground session.
At first, this seems like an easy "yes", but what if others' blueprints contain actual secrets they will later use in production?
Perhaps each export should have two files: an export zip and a separate sibling metadata file. That way the information is available but harder to accidentally leak to others
I think we would want Playground to be able to apply sensible defaults, as long as Playground doesn't modify the actual snapshot in-place.
This seems to be a place for separate-but-complementary tooling. Or maybe it is a different kind of export. |
Beta Was this translation helpful? Give feedback.
-
👍 But could we have a watcher for syncing these kinds of changes?
🚀 It's very cool to have db.php eliminated as a concern.
This was sort grappling with the idea of how much should be included and whether, instead of including copies of everything, an export could point to an external resource like
Maybe this should be configurable depending upon user intent. If these exports are just for Playground consumption, maybe we should just rely on Playground to do the replacements again when loading the export? Or if these are more generic for Site Transfer Protocol, then I suppose the export might just need to know what its current SITE_URL and HOME_URL are in order to make further replacements possible. I feel like maybe I'm failing to address the actual question here... (stepping away for now but planning to reply to the rest of your comment later) |
Beta Was this translation helpful? Give feedback.
-
nm... there wasn't much left. 👍 to everything else 😄 |
Beta Was this translation helpful? Give feedback.
-
Do we need to rewrite |
Beta Was this translation helpful? Give feedback.
-
|
I like this! It wouldn't work outside of Playground runtime, but then we don't need it to — moving those sites to regular PHP would an import flow where we could rewrite wp-config.php without the same consequences as here. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
I summarized some points in a new Boot protocol issue – thank you so much for all your great thoughts! |
Beta Was this translation helpful? Give feedback.
-
|
Actually I'm reopening the discussion. Here's one more conundrum to solve: In Playground docs workflow I have multiple Blueprints:
I wonder how could we reduce duplication between the web Blueprint and the development Blueprint. Unzipping an external zip is a part of a Blueprint, but mounting a local directory isn't. However, these operations correspond to each other. Also, with a "copy instead of mounting" feature, both could be expressed as Boot options, not Blueprint steps. I wonder:
Perhaps Blueprints could have a runtime-specific list of resources where runtime is web, CLI etc. Or maybe an "environment", e.g. "preview", "develop", "test" etc. would be a better abstraction? Something like this: {
"environments": {
"preview": {
"resources": {
"gutenberg": { "resource": "wp.org/plugins", "slug": "gutenberg" }
}
},
"develop": {
"resources": {
"gutenberg": { "resource": "local-mount", "path": "./gutenberg" }
}
}
},
"pathMapping": {
"/wordpress/wp-content/plugins/gutenberg": "gutenberg"
}
} |
Beta Was this translation helpful? Give feedback.
-
|
I see blueprints as automation and expect the site to be fully configured before they start running, so it would make sense to me that blueprints run only after mounting/copying. Also, we want blueprints to become part of WordPress in the future so I would make the decision based on if we want WordPress to support it or not.
The more we discuss this, the more I think it would be nice to have it. It can be an extended version of blueprints specific to Playground, or include |
Beta Was this translation helpful? Give feedback.
-
This idea does not smell right to me. I think it is because using "mount" to mean two different things indicates an imprecise solution language. In one case, "mount" means "mount", but in another place, we have to pre-arrange an interpretation to gloss over the conflict. "When I say 'mount', you say 'copy'." In addition, the different senses of the word "mount" have different orders of meaning. For JS runtimes, "mount" is more meta and actually defines the nature of "the ground beneath our feet" while, with the PHP runtime, "copy" just defines what rests on pre-existing ground. They are similar, but "mount" is platform/foundation language while "copy" is content language. Outer language describing the platform can naturally inform the content as it does in the case of "mount" (i.e., "mount a writable dir that already has some content"), but "copy" just describes content and doesn't say anything about the underlying nature of where the content lives. Please forgive me if I am belaboring this point. It is late, and I feel like there is a more articulate way to way to say all of this. :)
This seems better to me because I think we need different languages to articulately describe the platform and what rests upon it. They are related but not the same. My thinking is:
|
Beta Was this translation helpful? Give feedback.
-
All Playground apps need a consistent boot/export protocol. They don't have one yet. Playground.wordpress.net, Playground CLI, wp-now, Studio -- they all boot in their own way, creating inconsistencies and maintenance hurdles. Let's define a consistent boot/export protocol that Playground tools will be able to reuse.
What is a Boot/Export protocol?
There are two parts:
While there is an overlap with the Site Transfer protocol, this is a Playground-specific discussion.
Technical assumptions
/wordpressand/tmpand/internaldirectories in place. That's the case even in Node.js. We may eventually build tools that operate on the host filesystem directly -- they're out of scope for now as most of the problems outlined here just don't apply there.Discussion goals
Building VFS
Building VFS is the first boot stage. Once it's finished, we have the correct WordPress file structure. WordPress itself may not be able to start yet, since the database may not be configured yet, but all the files sources are at least wired to their correct location in VFS.
Building VFS involves:
Decisions to make in the flow -- let's discuss these points:
wp-content/index.php/wordpressdirectory/wordpressdirectory, hoping Playground will populate itwp-config.phprewrites to pollutegit status. We don't want the SQLitedb.phpto bleed out and get deployed to a repo containing a MySQL site.wp-config.phpmay need rewriting, e.g. when aSITE_URLconstant is already defined there. We could display a warning.db.phpfile must be created and we can't fake it./wordpresssunrise.phpfile could perhaps be faked as an mu-plugin.wp-content, say a few plugins, uploads but not a theme. How could that be automatically wired together in a correct way in a higher-level tool? Use-case: not keeping thetwentytwentyfourtheme in a GitHub repo just to make Playground happy.OPFSdb.phpis present because Playground is used to develop a drop-indb.phpplugin.playground.internalURLs be rewritten to the current server URL on boot in the imported files (not in the database)?Another visual to illustrate some of the challenges:
Out of scope for now:
Technical constraints:
auto_prepend_filedb.phpandsunrise.php.sunrise.phpcan maybe be avoided, but I don't think there's an easy way to avoid creatingdb.phpon the disk.Building the database
After this step, WordPress should be able to connect to the database and access the basic tables. In the 95% use-case it would already be a functional site. In 5% use-case, a custom "WordPress base snapshot" may rely on a Blueprint to finalize the setup and the site wouldn't work yet. In any case, the database connection and the required
wp_*tables should now be in place.Decisions to make in the flow -- let's discuss these points:
db.phpfile, if present, is related to SQLite?db.sqlitefile located?wp-content/mu-pluginsor in/internal?db.phpifwp-contentis mounted? (also decided in the Build VFS step)data.sqlfile to executedb.sqlitefile but are supposed to use MySQL?playground.internalURLs be rewritten in the imported data?playground.internalURLs be rewritten in the existing database?wp_*tables.Handle runtime changes
Running a Blueprint on a database-connected site may affect files in the mounted directories. Sometimes that's desired, sometimes it isn't.
Design conundrums -- let's discuss these points:
wp-content/plugins, but not the others, e.g. never change mywp-config.php?wordpress.zipfile a valid use-case? Or do will we always need an import tool to rewrite the site URL and, in general, apply the Site Transfer Protocol.Snapshot Export
Exporting Playgrounds is actually a part of the Boot process. Why? Because people will want to run those exports later on. Therefore we need to design them in a way that's nice and compatible.
Questions to answer
/wordpressdirectory with release-specificwp-includesandwp-admin?wp-config.phpbe exported?db.phpdrop-in be exported?/wordpress?/wordpress/wp-content? Should that include customized/wordpress/wp-config.php?/wordpress/wp-content/plugins/my-plugin?/wordpress/index2.php? Should Playground detect that / care?/internaldirectory?/internal/mu-plugins/sqlite-database-integration? It's referenced in/wordpress/wp-content/db.php.Blueprint.jsonwith WordPress, PHP versions and env/constants included in the export?metadata.jsonwith{"oldUrl": "https://mysite.com", "requiresWp": "6.4"}included in the export? Or is this handled with a Blueprint?https://playground.internal?/-based URLs?Other questions
cc @bgrgicak @dmsnell @brandonpayton
Beta Was this translation helpful? Give feedback.
All reactions