Consider Transforming this into a Monorepo after the initial Website Redesing is done

[ovflowd]

After we get the Website Redesign done, we might want to change to a Monorepo structure, which could quickly offload which parts of the Website need to be built due to what is being changed. This should be considered only after we successfully completed #5403.

What is this about?

• An efficient rollout process where specific GitHub Actions are triggered only for relevant "domains" (separation of domain/concern)
• No need to build the whole Next.js website for every change
• Tests and other Workflow steps of different parts only happen when needed

How would the Monorepo be structured

• Very possibly with NPM workspaces or PNPM monorepo
• It would contain the following packages:
  • packages/i18n (versioned)
    • Internationalisation Engine
    • Internationalisation Data and Translations
    • Unit Tests
  • packages/ui-components (versioned)
    • Contains all Components that currently live on components
    • And also all Hooks that are only used by components
    • Storybook and Storybook Tests
  • packages/core (Name subject to change) (versioned)
    • All React Providers, Hooks, Shared Types, and Utilities that are used across other packages
    • Unit Tests
  • apps/nodejs-website
    • Next.js Pages, Middlewares, Layouts, Routing, API Routes, etc

cc @nodejs/website as I'd love to get opinions on this matter from y'all :)

[umairraza96]

What about turborepo or nx those are also phenomenal choices for monorepo

[ovflowd]

We do use turborepo :)

[ovflowd]

@Harkunwar, I guess it would make sense to start investigating the Monorepo approach.

Things to keep in consideration:

• packages structure as mentioned above
• When on Pull Requests, suppose we have shared scripts (3 packages have tests, for example)
  • Only the modified ones have their tests to run?
  • Run lint-staged for linting cross mono repo?
• Find a way that we don't need to rebuild already up-to-date packages? (I guess turborepo will help on that, but we need to see how to configure turborepo on a monorepo for each package)

Possible green flags:

• Build time is reduced due to fewer packages
• Overall folder structure feels cleaner
• Way fewer files and folders directly on the root directory
• Easier to understand what belongs where

Since on the infrastructure side of things, things look good enough, and the work we have now is for migrating components (at the moment), it might be a good bet of migrating this to a monorepo.

[Harkunwar]

@ovflowd Sounds good to me! I'll start investigating migrating to monorepo.

[AugustinMauroy]

Hey @Harkunwar! I'm currently testing monorepo for this project. Where do you are in this process ?
here my "POC"

[AlexcastroDev]

Here my 2 cents:

NX preach that the project should have 1 package.json, and versions should be the same for all projects.

The tradeoff is when you have 2 or 3 Apps, and a Storybook, you will always download the app dependencies with storybook as well. In the case you use npm install --prod, should be a better way to use on CI.

Turborepo allows you to have many package.json for projects. It can be good or bad haha. I have been using Turborepo in the company i work, for mantain 4 apps, and some components library.

But also i use NX in other repository, because it already have the bundler with near zero configuration. That's a good approach for TS / JS Packages. We use this for service layer and analytics.

We have a good point to use NX for our storybook library.

In summary, Turborepo is good for Many Apps, and components library.
But NX is better for packaging managment and uniform libraries

---

ChatGPT correct my poor english:

When it comes to project management practices, NX advocates for a single package.json file and consistent versioning across all projects. However, there's a notable consideration to keep in mind. In scenarios where you're dealing with multiple applications (2 or 3) alongside a Storybook, a potential tradeoff arises. When you perform an npm install --prod, the app dependencies and Storybook are downloaded together. To address this, finding a more efficient way to use it within a Continuous Integration (CI) context becomes important.

This is where Turborepo comes into play. It offers the flexibility of employing multiple package.json files for different projects. The outcome can be positive or negative, leading to a mix of experiences (which can be quite amusing!). Personally, I've leveraged Turborepo within my company, managing four applications and a components library. On the flip side, for separate cases, I've utilized NX in other repositories. The reason being, NX provides an almost effortless configuration for bundling, especially ideal for TypeScript (TS) and JavaScript (JS) packages. This approach works remarkably well for our service layer and analytics needs. Additionally, NX shines as a great choice for managing our Storybook library effectively.

In a nutshell, Turborepo proves its worth when dealing with numerous applications and a components library. On the contrary, NX emerges as the preferable option for streamlining packaging and maintaining consistent libraries across the board.

[ovflowd]

Hey @Harkunwar! I'm currently testing monorepo for this project. Where do you are in this process ? here my "POC"

Looks cool. Except that api-docs are not the scope of this repository. We do aim to have a package called design-system but I would rather have it called ui-components So then we can publish it under @nodejs/ui-components

Note: With publish I mean locally within the monorepo, we're not going to publish these on NPM.

[ovflowd]

Hey @AlexcastroDev I appreciate your input here. We're very probably sticking with Turborepo as it is what makes sense here.

Mostly because we rely on Next.js;

I never heard about nx and not sure what exactly it brings to the table. Turborepo is also extremely simple. For bundling we can very probably going to use parcel since the nature of each package is extremely simple.

For example, tie i18n package will only at most have a bunch of JSON files, the ui-components will have just pure React Components.

[AugustinMauroy]

Note: With publish I mean locally within the monorepo, we're not going to publish these on NPM.

I don't understand this part. Not the sentences I mean the code logic behind.

some question to advance on this issue

• do we need to compile UI lib ? Because if we just keep the .tsx file and import them I nextjs app it's work and reduce the logic behind.
• (i shaw you removed "api" form the comment) so where it's should be done ?

[ovflowd]

I don't understand this part. Not the sentences I mean the code logic behind.

Hm? Could you explain?

do we need to compile UI lib ? Because if we just keep the .tsx file and import them I nextjs app it's work and reduce the logic behind.

Yes. The UI Component Library needs to be transpiled (TypeScript -> JavaScript) and then minified for Production usage. It also needs to be Framework agnostic (e.g.: Not Next.js nor React dependent) so it will use plain JSX as a target.

[AugustinMauroy]

Hm? Could you explain?

You say "publish" but also "not publish to npm". So what do you mean by publish

[ovflowd]

Hm? Could you explain?

You say "publish" but also "not publish to npm". So what do you mean by publish

Pretty much "publish" on the repository, through git tags and GitHub release feature :)

[AugustinMauroy]

I think we'll be able to do this issue after the Redesign is online.

[anthonyshew]

👋 Here from Turborepo team. Happy to help as needed!

[ovflowd]

👋 Here from Turborepo team. Happy to help as needed!

Any help is appreciated! Feel free to join our Slack if you'd like to coordinate this!

[anthonyshew]

@ovflowd I'm seeing two active Slack servers on the Get Involved page. Do you have a recommendation for which one to hop into and a channel(s) to offer up support in?

[AugustinMauroy]

@anthonyshew this link is normally valid

[ovflowd]

@ovflowd I'm seeing two active Slack servers on the Get Involved page. Do you have a recommendation for which one to hop into and a channel(s) to offer up support in?

Node Slackers is not an official community maintained by the Foundation, so only the Foundation Slack matters

[anthonyshew]

Thanks, both of you! Will join the party.

[ovflowd]

@anthonyshew how can we get this started? :) would love to start coordinating efforts to make the monorepo as described on the opening issue possible!

[ovflowd]

Hey @anthonyshew any progress? Is there anything we can help you to get started?

[anthonyshew]

@ovflowd Hey, sorry for not seeing these pings! My GitHub notifications are...effectively useless. 😄

Catching up with the conversation here, it looks like I'm in general agreement with the approach described (with some slight tweaks). Before I write out what my recommendations would be, a couple questions:

• I'm seeing the library packages being marked with a (versioned). Are these packages going to be published to the npm registry? If not, they likely don't need to be versioned, since versioning for Internal Packages serves nearly no purpose, while adding overhead.
  • There are a few other purposes I've seen for versioning internal-only library packages but they're highly situational. One of those situations may exist here and I'm not aware!
• Is there intent to add more than nodejs.org to this repository in the future?
  • This question is mostly curiosity - but the answer has potential to push things toward "definitely monorepo" if it's yes.
• Is there a discrete list of pros/cons that we should draw out? I ask because, in the Issue description, I see more "how" than "why". As someone not familiar with this repo, I'd be interested in hearing thoughts about things like:
  • While domain separation can be done using a multi-package workspace, could similar partitioning be achieved with a meaningful directories in a single-package workspace (the current structure)? Does the overhead of a multi-package workspace add value above using directories when it comes to creating domains?
  • Is the repository at a scale where the overhead is justified? If it isn't today, will it be in the foreseeable future? Are the durations of tasks being ran (when they miss cache) hurting shipping velocity?
  • Does using a multi-package workspace in this repository unlock specific capabilities or workflows that are not possible today? I see the mention of breaking up tests, lints, etc. Are there more?
  • Other considerations? It's likely this isn't an exhaustive list. There could be more to think about!

None of this is to say that I'm against converting to a multi-package workspace. I'm moreso making sure we did our homework before making this change. 😄

So, assuming we continue towards a monorepo, a few things that I'd be recommending:

• I see pnpm was mentioned and that would be my recommendation if there is flexibility in package managers. pnpm has more monorepo features and is generally more correct about dependency handling than the other major package managers.
  • Of course, npm can still be used to keep things closer to traditional Node.js. Totally valid choice to make here in Node.js-land!
• Storybook would need to be moved to an application rather than into the UI library.
  • You want a clear separation between "where are deployable apps" and "where are shared libraries" in the workspace so that you have a more accurate dependency tree to control deployment behavior. Additionally, importing those UI components across a package boundary more closely mimics how you would import them into the production applications.
• I would recommend an incremental strategy to get from today's structure to the eventual structure in the Issue description. This helps to de-risk the migration, minimizing variables for each step. Roughly, the steps would be:
  1. Move the existing repository into ./apps/web (or similar) and create a new repository root with a new package.json, turbo.json, etc. It's likely some re-alignment of deployment settings for infrastructure and CI may be needed.
  2. Create config directories to hold shared configurations for things like ESLint, TypeScript, etc.
  3. Create libraries, one at a time. Depending on how defensive maintainers would like to be, PRs to create libraries could be done all at once (create package directory and move code) or incrementally (create package directory, move code later).

There's a lot more to how that migration works but I've likely already written too much here so I'll take a breath to let others respond. 😄

[bmuenzenmeyer]

@anthonyshew

• I see pnpm was mentioned and that would be my recommendation if there is flexibility in package managers

pnpm has been discussed but there is a bit of concern about contributor approachability #5334

yes we all know about corepack, but we REALLY care about limiting tooling, even with how easy corepack enable && corepack install would be. We get so many casual and/or busy folks heading this way that the default might not be the headache of additional overhead

Is there intent to add more than nodejs.org to this repository in the future?

Agree some of the breakdown in the OP looks like directories, without any clear reasoning as to why a discrete versioned/published package is necessary. To be fair, the OP content is now quite dated and might not reflect the state of our goals or the tech.

This is largely in support of helping the api docs get a facelift. IIRC, the goal is to extract the styles out into a pacakge so we can import them into the main nodejs/node project. how we do that is not yet resolved.

nodejs/node#52343

[ovflowd]

I'm seeing the library packages being marked with a (versioned). Are these packages going to be published to the npm registry? If not, they likely don't need to be versioned, since versioning for Internal Packages serves nearly no purpose, while adding overhead.

Yes, all the UI components are one because we want to be able to use these UI components in other repositories. Is a publish needed? I assume we can reference the package through NPM's git resolution algorithm? i.e., "nodejs/web-components": "git://github.com/nodejs/nodejs.org/packages/...." (Not even sure this would work as expected)

Is there intent to add more than nodejs.org to this repository in the future?
This question is mostly curiosity - but the answer has potential to push things toward "definitely monorepo" if it's yes.

This repository? No.

While domain separation can be done using a multi-package workspace, could similar partitioning be achieved with a meaningful directories in a single-package workspace (the current structure)? Does the overhead of a multi-package workspace add value above using directories when it comes to creating domains?

We want to simplify CI processes and decouple dependencies to make build, testing, rollbacking, and maintenance easier. We also want to be able to use certain packages outside of the nodejs.org repository in other projects and packages.

I see pnpm was mentioned and that would be my recommendation if there is flexibility in package managers. pnpm has more monorepo features and is generally more correct about dependency handling than the other major package managers.

I'd recommend that everything is NPM compatible and recommended by default, but our CI could use PNPM. We would still recommend NPM as the default method for newcomers, but we can use PNPM within our CI to speed up CI build, processes, etc.

You want a clear separation between "where are deployable apps" and "where are shared libraries" in the workspace so that you have a more accurate dependency tree to control deployment behavior. Additionally, importing those UI components across a package boundary more closely mimics how you would import them into the production applications.

Solid point.

[ovflowd]

Agree some of the breakdown in the OP looks like directories, without any clear reasoning as to why a discrete versioned/published package is necessary. To be fair, the OP content is now quite dated and might not reflect the state of our goals or the tech.

If no versioning is needed besides of just Git commit versioning, I'm fine with it :)

Since most of packages would be consumed internally and not by outer parties/community.

[anthonyshew]

Heard to both of you! It sounds like there's been much more thought put to making this move than I'm aware of or could find. Awesome!

I'm still unclear on the publishing story - but sorting that out can be done after creating the multi-package workspace's structure.

Thinking about migrating now, does this plan sound like the approach we'd like to take?

1. Move the existing repository into ./apps/web (or similar) and create a new repository root with a new package.json, turbo.json, etc. It's likely some re-alignment of deployment settings for infrastructure and CI may be needed.
2. Create config directories to hold shared configurations for things like ESLint, TypeScript, etc.
3. Create libraries, one at a time. Depending on how defensive maintainers would like to be, PRs to create libraries could be done all at once (create package directory and move code) or incrementally (create package directory, move code later).

[ovflowd]

I'm still unclear on the publishing story - but sorting that out can be done after creating the multi-package workspace's structure.

If we can fulfil what we need without publishing, I'm down for it :)

[ovflowd]

Thinking about migrating now, does this plan sound like the approach we'd like to take?

Sound like a solid plan!