Update repo structure, package structure, and release processes for 1.0

[danielroney]

This epic will address the high level plans for the repo and package updates we want to make before we declare 1.0.

Attach sub issues here as a task list.

First step is to layout the plan and phases.

This work is codependent with #8008, where the design decisions of this item impact the work needed for #8008 and where this item also encompasses the integration of #8008. There is also additional related work tracked by #8945 that covers items post 1.0.

Due to a change in views around the meaning of 1.0, some goals have been shuffled around. Particularly, the priority of being more intentional about breaking changes and the related infra has increased, while the priority of fully independent versioning has decreased.

Goals

• Our packages are in the same repository (as opposed to the current split repos of client, server, common-definitions, etc.) Moved to post 1.0
• Packages are versioned independently. Packages may be assigned to groups which are versioned in lockstep Moved to post 1.0
• Developers can easily determine what versions would need to be released at any time
• Developers can create pre-release builds, and pre-release versioning is understandable
• Releases happen in a single user operation (as opposed to the current process of doing a release for each layer) Moved to post 1.0
• Developers can work easily across multiple development branches that permit different levels of breaking changes. This involves setting up parallel dev branches for changes that are not allowed on the primary dev branch (e.g. due to the level of breaking changes permitted at any time) and the related processes and dev education to support this. This is expected to involve creating a branch for major changes that must wait until the next major release cycle while the main branch can only make minor releases.

Work items

Developers must be intentional about breaking changes

• Add infra to reactively bump package versions #8054
• Add version/release sentinel file that tracks release branches #8458

Developers can work easily across multiple active development branches (main/next)

April

• Automate change integration from current version to next version dev branches #9296 ➡️ May

May

• Automate change integration from current version to next version dev branches #9296

Developers understand what versions are released

Next

• Generate version information about releases for consuming developers #8946

Additional Pre 1.0 Items

• Add support for virtual patch versioning #9292
• Update release process instructions for virtual patch and 1.0 transition #9293
• Update pipeline definitions to account for new active development branches #9367
• Update public documentation to reflect new breaking change handling #9390

April

• ~~ Integrate the automated API change detection into PR validation #8028 *This is an item in the API Breaking change detection #8008 epic as well. Not all items in that epic to bring this validation to ~100% are necessary here, but it should be ~100% for the definitions layers/everything below server. ~~ ➡️ May
• Separate "3P" packages into their own group separate from the rest of "client" for 1.0 #9294 ➡️ May

May

• Integrate the automated API change detection into PR validation #8028 *This is an item in the API Breaking change detection #8008 epic as well. Not all items in that epic to bring this validation to ~100% are necessary here, but it should be ~100% for the definitions layers/everything below server.
• Separate "3P" packages into their own group separate from the rest of "client" for 1.0 #9294
• Update fluid-build to account for azure release group (monorepo) #10221
• Add build/CI pipelines for azure release group (monorepo) #10222
• Integrate azure build pipeline with downstream pipelines (e.g. docs) #10223
• Update CODEOWNERS with azure project paths and owners #10227

[anthony-murphy]

The overall issues here look pretty good, but i think one important aspect is missing, and that is how the repo should work outside of release.

I think the main branch should be fully linked, as in all packages always build together, and none of our packages are pulled from npm. This is quite different from how it works today. Today in main, our common, server, and client packages do not form a consistent set, specifically the version of common and/or server the client packages depend on is usually that of some prior release. Only when we wish to release client packages with an updated common or server package do we bump all the version and link the repo.

This brings me to my second point, which is why we did it. We did it that way to reduce churn in packages that have no, or only minor changes. This gives our users a more stable set of packages where we only do a version bump if there are necessary changes. This is something we would like to keep. Today we only really have a two part version. Technically the two parts we have are minor, and patch, however npm/semver has funky behavior when the major version is 0, which is what ours is, 0.minor.patch. This leads to the minor version acting like a major version. This is part of the problem we face. The second part is that we need to reserve the patch version, otherwise we have no space in the version to make fixes in previously releases. Because of these, we bump minor version at release time, and we don't want to immediately consume that for the reasons above, it leads to churn.

To have a fully linked repo, and to avoid version churn, we need a three part version, major, minor, and patch. Patch serves the same purpose as today; it is for fixes in prior releases. Minor is the version we should bump by default after release, this can technically be done for all packages, as minor versions per semver are compatible. Major should only be bumped when necessary, aka there is a breaking change, which we have other workstreams to help detect.

Once we reach 1.0.0, i think this is straightforward, as we gain access to the missing major version, but we also need a solution for today, where our version starts with 0. I see two possibilities here, somehow leverage pre-release versions, 0.0.0-0 to gain the missing version part. However, i have concerns about out ability to maintain compatibility with semver, npm, and related tooling, as we cannot just shift to 0.major.minor-patch as everything after the dash is treated very differently, and generally, prelease builds should not be consumed for productions. Because of the issues i see around doing this i'm not sure the added complexity is worth it, especially as this will only be necessary in the interim before 1.0.0. What i believe is a simpler scheme is to overload the patch version. To do this i think we should bump our patch version by some large constant like1000 at release. That constant reserves that version space for true patches to prior release. For instance if we were to release a package at 0.2.0, we would bump the version in the repo to 0.2.1000. This would allow us to make up to 1000 patch release before we conflict with the next releases version space. Essentially, the most significant digit of the "patch" version is the minor version, and the rest are the true patch version. While the version numbers in this scheme may look a bit odd to humans, they should work seamlessly with all tool based on semver and npm.

[heliocliu]

@anthony-murphy Agree about the full linkage. #8026 (under the other epic #8008), tracks demonstrating it. I should have created another item here covering the actual merging of the repos.

I think any decisions the versioning point falls under #8054, but I disagree with continuing our current practice of bumping anything right after release. Assuming 3 part versions, preemptively bumping minor makes branch versions increasingly confusing over time and seems unnecessary because we need the change detection tooling anyway. If we always bump minor, what happens when it's time to bump again if there are no changes? Regardless of if there's another bump, some branches will have packages versions that were never published and it's not possible to tell except by looking at the actual release list. When in a release cycle that permits major version changes, we can't preempt a major bump, and packages that end up needing a major bump will need a change to fix those versions - in which case we could to the minor bump in the same way. This also violates semver expectations (though I don't think it really matters in practice).

The external version management tools we looked at before (lerna/yarn) both view the package version in the repo as meaningless because the increment can't be determined in advance, and the value only gains meaning when in the published package. I think it makes sense for us to do the same - separately track what needs to be incremented (e.g. as a centralized version list that is updated alongside breaking changes) and apply all those changes at once as part of the branching/publishing steps, but otherwise don't touch the version in the package.json (this can be fudged a bit to support prereleases/betas). This still lets us reduce package version churn, we skip updating what will frequently be a meaningless number, and the separate tracking lets developers see the state of expected version changes.

[anthony-murphy]

This sounds reasonable to me. I really don't have a strong opinion of around if we pre-bump or not, or whether we bump packages without changes or not. The key points are that we should always have full linkage, and maintain room in our version space for patches to prior releases. You also bring up a good point around pre-release/betas.

I would love to see a design in this issue that describes how this will work, and too get that reviewed.