[RFC] Basic Deployment Solution For Modern.js

[yimingjfe]

Summary

The purpose of this proposal is to provide deployment support for Modern.js.

This RFC is a basic solution for deployment, and solutions for different platforms and runtime environments will be extended based on this solution.

Usage

We will provide a new deploy command:

Deployment platforms usually use the build command as a convention, and I also think that only the build command is enough, but in order to avoid a break change, we add a new command here.

npx modern deploy

When this command is executed, the framework will detect the deployment platform and outputs the corresponding production output.

Also, developers can specify the currently deployed platform by an environment variable:

MODERNJS_DEPLOY=node pnpm deploy

Multi-platform Support

As shown in the figure, we will continue to adjust the server-related code to make it platform-independent as much as possible, for different runtime environments, the framework will generate different entries for adaptation.

Structure Of Production Output

As you can see below, index.js is the generated entry for the different platforms.

In the actual scenario, the directory structure will be slightly adjusted for different deployment platforms.

.output
├── static
├── node_modules
├── bundles
├── api
├── server
└── index.js

Production Dependencies

We have a deployment solution in the company, but the handling of dependencies is challenging for users, with the following issues:

• After the build, users need to install dependencies in a shell script, which is not friendly.
• Because @modern-js/prod-server is not a direct dependency of the project, we had to add a phantom dependency.
• It is impossible to ensure that the dependency is locked for monorepo.
• For different monorepo solutions (such as pnpm, rush), we need to make different adaptations.

There are some ways to handle dependencies for deployment:

1. Bundle all dependencies, but for us, our server code and dependencies are not yet ready for this.( we can use rsbuild to bundle our server in the future, but there will still be some dependencies that need to be external.)
2. Use the deploy solution provided by the monorepo tools(such as pnpm, rush), but there are the following problems:
   1. We need to adapt and provide best practices for different monorepo tools, users also need to learn these practices.
   2. Modern.js is a full-stack framework with both front-end and server-side dependencies in the project , in such a way that there are still front-end dependencies in our production output of server.
3. Reinstall dependencies by package manager in the production output directory.
4. Trace the dependencies used in the server-side code, and rearrange the dependencies in the node_modules directory of the output by the framework(by copying and creating symbolic links).

In the past, we used a combination of the second and third approaches and pruned the dependencies, which brought up the above problems, so I prefer the last solution.

The following is a basic example of the last approach:

• bar is a dependency of foo, and there are multiple versions of the bar in the project.
• utils is a workspace package that is depended on by the currently deployed project.
• -> means a symbolic link

node_modules
├── .modernjs
│   ├── bar@1
│   ├── bar@2
│   └── utils-dep1
│       └── node_modules
│           └── utils-dep1
├── bar -> .modernjs/bar@2/node_modules/bar (link)
├── foo
│   └── node_modules
│       ├── foo
│       └── bar -> .modernjs/bar@1 (link)
├── utils (workspace package)
└── utils-dep1 -> .modernjs/utils-dep1/node_modules/utils-dep1 (link)

Details

Compared with package managers, our scenario can be much simpler, so we don't need to write a package manager, here are the specific ways:

1. Firstly, the framework will trace all dependencies.

2. For the handling of dependencies, the framework will follow the following rules:

   • For single-version dependencies, copy the package to the node_modules directory.
   • For multiple versions of dependencies, copy all packages to the node_modules/.modernjs directory, the newer version of package will be hoisted to top level of node_modules and and each package will be created a symbolic link to its parent's node_modules.
   • For workspace packages, copy them to the node_modules directory uniformly. For dependencies of workspace packages, handle them according to the rules above.

3. Finally, generate a new package.json based on the original package.json and dependency information.

Limitations

1. In the server-side code, commands provided by the package cannot be executed, because we don't handle the .bin directory.
2. We analyze dependencies based on ast (using @vercel/nft), and there are some dynamically required packages that we can't resolve, so we need to provide a configuration to include some dependencies in later version.

[zllkjc]

Will we build platform-related code into the framework or provide plugins?

[yimingjfe]

It will be in our framework now, because it doesn't require any additional dependencies.

[zllkjc]

the logic in which package, app-tools ?

[yimingjfe]

Yeah, it's in the app-tools, because the deploy should be a default feature.

[zllkjc]

output
├── static
└── server
    ├── bundles
    ├── api
    ├── server
    └── index.js

Where is the html files, in the server/server/html ?

[yimingjfe]

Yeah, the file is not mentioned in detail in this RFC, which will be consistent with the structure in the dist directory.

[zllkjc]

So, when we exec deploy command, node_modules/ will be included in the output dir ?

[yimingjfe]

Yes, all required dependencies will be in the output directory.

[zllkjc]

What is the architecture of this logic and in-house framework? Is it simply set for the deploy command of modern.js?

[zllkjc]

Why not just bundle the @modern-js/prod-server, and let developer to deal the other dependencies which declared in the project package.json

[zllkjc]

The only benefit is that the user's dependency size reduce...but I think this should still be done at the step of recover dependencies. For example, Modern.js can output a real JSON of dependencies that are actually used in the server code, allowing the next process of recover dependencies to handle it

[zllkjc]

Or at least we should not provide this dependency recovery capability to Monorepo. For singleRepo projects, it can indeed reduce a lot of their work.

[yimingjfe]

I think we should not interfere with the logic at the Monorepo level. Each Monorepo is likely to have its own practice, and our approach may not be what they want. We should provide less ability..

I understand your concerns, but if we want the deployment to be as simple as possible for users and don't need to provide different practices for different monorepo tools, we need to do this.

In the long term, our goal is to bundle dependencies as much as possible, it also don't need monorepo tools to recover dependencies.

[yimingjfe]

The only reason why we need to adapt to different monorepos is the phantom dependency problem of packages such as @modern-js/prod-server, because different Monorepos have different ways of recover the dependencies

It's not just our dependencies that need to be restored in the deployment, but also the server-side dependencies of the project.

[yimingjfe]

Moreover, I think if we do bundle @modern-js/prod-server, there will be no "difficult to lock the versions", because all the other dependencies are in user package.json..and user can lock by lockfile...

Fully agree that we should bundle the entire server, here are the indirect dependencies used by the project that we can't lock, we've encounter this problem in previous practice, for example when the monorepo use the infra directory.

[zllkjc]

OK that great..So the conclusion:

• Modern.js do dependencies by ourself even in monorepo
• After that we should support the bundle for server code
• Maybe we need distinguish bundle 'prod-server' and recover other deps by monorepo in more scenarios, such as monorepo's unified infrastructure

[yimingjfe]

After the discussion, to add to the third point, our ultimate goal is to bundle the entire server, the dependencies that are external will be handled using the dependency handling in this RFC, if it doesn't work well with some internal projects, we are considering to provide the configuration, only bundle prod-server, and not by the framework layer to do the recover deps