RFC - Semver redesign

[yjaaidi]

Context

The current version of semver (2.x) has some known limitations mainly because we designed it as an Nx executor.
Here are some examples:

• performance issues as we have to run the executor on each project when running in independent mode
• executor is hard to parallelize due to concurrent access to git
• grouping commits is harder

The other major issue is that development workflows and project structures can vary a lot between workspaces. Thus, grouping and versioning strategies can vary a lot. Providing multiple options to cover all different use cases increases the surface of semver and can even make it confusing or false feature-rich.

Goals

In order to fix the issues above, semver 3 will be designed with the following goals:

1. semver should run once on the whole workspace as a standalone script instead of a project executor: yarn semver or yarn nx semver.
2. semver should allow extension using custom strategy implementations (e.g. semver.config.ts) instead of options.

sequenceDiagram
  Note over Core: 1. resolve strategy
  Core->>Core: resolveStrategy(): Strategy

  Note over Core: 2. build versionable tree based on nx dep graph

  Core->>Core: getNxProjects()

  Core->>Strategy: resolveVersionables(projects:  NxProject[]): VersionableInfo[]

  Note over Core,Strategy: resolve tag prefix for each versionable
  Core->>Strategy: resolveTagPrefix(versionable: VersionableInfo): string

  Note over Core: 3. resolve last version for each versionable
  Core->>Core: resolveLastVersion(tagPrefix: string): Version

  Note over Core: 4. resolve changes (commits + deps commits)
  Core->>Core: resolveChanges(paths: string[], since: string): Changes

  Note over Core: 5. Build versionable tree based on dep graph
  Core->>Core: resolveDependencies(versionableInfos: VersionableInfo[]): VersionableInfo & {deps: VersionableInfo[]}

  Note over Core: Group everything in Versionable object

  Note over Core,Strategy: bump
  Core->>Strategy: bump(...)

  Note over Core,Strategy: update files
  Core->>Strategy: updateFiles(...)

  Note over Core: commit 
  Core->>Strategy: commit(...)

  Note over Core: finalize
  Core->>Strategy: finalize(...)

classDiagram

VersionableNode o-- VersionableNode
VersionableInfo <|-- Versionable
Versionable <|-- VersionableNode


note for NxProject "all these properties are used by the strategy\n to group projects into versionables"

class NxProject {
  type: 'app' | 'lib';
  name: string;
  path: string;
  tags: string[];
}

class VersionableInfo {
  name: string;
  paths: string[];
}

class Versionable {
  changes: Change[];
  dependencies: Versionable[];
  tagPrefix: string;
  version: Version;
}

---

Raw draft notes

strategy = resolveConfig();

semver.getProjects(); 
// [{name: 'a', path: 'apps/a'}, {name: 'a-ui', path: 'libs/a/ui', tags: ...}, {name: 'x', path: 'libs/x'}]
      ||
      \/
strategy.resolveVersionables();
      ||
      \/
class Versionable {
  name: string;
  paths: string[];
}
// [{name: 'a', paths: ['apps/a', 'libs/a/ui']}, {name: 'x', paths: ['libs/x']}]
      ||
      \/
semver.buildGraph(); 
      ||
      \/
class VersionableWithDeps {
  name: string;
  paths: string[];
  deps: VersionableWithDeps[];
}
      ||
      \/
strategy.resolveTagPrefix()
      ||
      \/
class VersionableWithDeps+TagPrefix {
  name: string;
  paths: string[];
  deps: VersionableWithDeps[];
  tagPrefix: string;
}
      ||
      \/
semver.resolveLastVersion()
      ||
      \/
class {
  name: string;
  paths: string[];
  deps: VersionableWithDeps[];
  tagPrefix: string;
  version: string;
}
      ||
      \/
semver.computeChanges()
      ||
      \/
class {
  name: string;
  paths: string[];
  deps: Versionable...[];
  tagPrefix: string;
  version: string;
  commits: Commit[];
}

Resolve groups

interface Versionable {
  name: string;
  paths: string[];
  publishable: boolean; // ignore this for now
  changes: Changes[];
  deps: Versionable[];
}

type GroupResoverStrategy = (workspace: Workspace) => Versionable[];

// ex. independent
const independentStrategy: GroupResoverStrategy = (workspace) => {
  return workspace.getProjects();
}

// ex. sync
const syncStrategy: GroupResoverStrategy () => { return {name: 'my-workspace', path: '/'} }

// ex. group by nx tag
// workspace: 
// - apps/a (nx tag: scope:a)
// - apps/b (nx tag: scope:b)
// - libs/a/ui (nx tag: scope:a)
// - libs/a/core ...
groupByNxTag('scope')(workspace); // => [{name: 'a', paths: ['apps/a', 'libs/a/ui']}, {name: 'b', paths: ['apps/b']}]

Resolve tag prefix

type TagPrefixResolver = (versionable: Versionable) => string;

Resolve last version

git tag -l 'semver-*' --sort=-v:refname | head -1

Build graph

TODO

Filter deps

The default implementation of this step is filtering all publishable deps.

Given: A => B publishable & C

Then graph would be: A => C

Compute changes

Given the following versionables:

• a: apps/a, libs/a/ui
• x: libs/x
• y: libs/y

& nx dep graph is apps/a => libs/a/ui => libs/x => libs/y

When a breaking change happens on y

Then

getChanges(a); // 
getChanges(x); // 1.0.0 => 1.0.1
getChanges(y); // {commits: [{type: 'breaking change', message: 'xxx'}]}

function bump(versionable) {
  const commits = getCommits(versionable.name);
  const depsChanges = getDeps(versionable.name).reduce((dep, acc) => ({...acc, [dep.name]: computeBumpVersion(...)}), {});
  const newVersion = computeBump(versionable, commits, depsChanges);
  return {
    version: newVersion
  }
}

[jbadeau]

What’s the timeframe for this and how can I help?

We are currently happy using semvers for our trunk based flows but would love to expand to other flows. Would be interested to see how v3 would help.

[edbzn]

It's too early to promise any landing date. There are multiple things you can do to contribute. You can give your thoughts about this RFC, and provide any other ideas you might have to improve semver. The reason behind this RFC is to collect ideas from our users and publicly design the new version, so if you need anything in particular that is not currently handled in semver v2 it's the right place and the right time. You can also open PRs against #651 to help moving forward in the implementation, no real progress was made from our side, so you can implement almost what you want as far as it's following the RFC, any help in development is very appreciated, and we'll help you to stay on track (using draft PRs, discussions, issues, whatever). Don't be shy to open any discussion to make everything clear. Finally, you can share this RFC with anyone who could have valuable input.

[mpsanchis]

Hi, I just opened issue #688 regarding wrong version calculation when using --releaseAs prerelease, but I saw now that you guys are working on a new version.
Basically my issue is a summary of 4 other issues and a discussion, all of which pointing to this bug/unexpected behaviour of semver.

Is this a goal to tackle in v3.0.0? Or is it out of scope?

Thanks a lot for the work put in

[yjaaidi]

Thanks @mpsanchis for your feedback on this.
This is in fact something that we didn't discuss yet concerning semver 3 and it's good that you raised the issue.

We have to include this in the discussion. If you have any suggestions on how you'd like to see this implemented with the new strategy system, then go ahead 😊

[yjaaidi]

In my opinion, the cool thing here is that this is somehow implicitly solved with the new strategy system as one can override the bump function.

Of course, we will have to provide the primitives that help customizing the bump.

We will also have to provide a way of overriding the getLastVersion function so it probably should be part of the strategy.

While this could be an option like --prerelease, I think that it's better if it's part of the strategy.
In fact, some might want to mark a library as a prerelease while other libs are regular bumps.

[mpsanchis]

@yjaaidi @edbzn: I sent you an email with some ideas/suggestions. Let me know when you have time to discuss it 😊

[edbzn]

Hi @mpsanchis, thanks for reaching out! I will answer here so it will be easier for everybody to follow. I think it would be nice to discuss everything related to the design here, so we can keep track of the propositions with one single source of truth.

I follow some of the suggestions you mentioned in the mail, but I think you mess one important point in this redesign: we identified two parts, core and strategy, the strategy is fully customizable by the user. We will provide some default strategies but users can create their own strategies based on what they need. We also don't want to create an Nx plugin, semver will only be a regular node.js script.

In the mail, you also mentioned a set of customizable lifecycles called plugins. That makes me think about semantic-release, the idea is interesting, but then we need to clarify the differences between the concept of strategy.

We are definitely up for discussing and collaborating on the next version, this project is open and we value any contributions.

[yjaaidi]

BTW, while this won't be an Nx plugin anymore, we will still rely on Nx to build the dependency graph and analyze affected projects.

[mpsanchis]

Hi team,

Replying to @edbzn’s point first:

• I mentioned semantic-release in the email, since we did go through it and liked the approach :) I hope we can also have a lean tool that can be easily extended.
• You’re right, I didn’t mention the “Strategy” directly. However, it seems like our idea of “plugins” would just be the “Strategy” split into smaller pieces.

So, as an overview, we would see the interaction Core-Strategy like this (it’s just a very rough draft):

sequenceDiagram
  box Core
  Participant Core as Core
  end
  box Strategy (group of plugins)
  Participant Pre1 as filter-is-versionable (PRE)
  Participant Calculate1 as calculate-semantic-version (CALC)
  Participant Process1 as some-processor (PROC)
  Participant Post1 as export-to-envvars (POST)
  Participant Post2 as tag-repo (POST)
  end
  
  Note over Core: 0. build versionable tree based on nx dep graph

  Core->>Core: getNxProjects()

  Note over Core: 1. PRE
  Note over Core: Call all pre plugins

  Core->>Pre1: resolveVersionables(projects:  NxProject[]): VersionableInfo[]

  Note over Core: 2. CALCULATE VERSIONS (CALC)
  Note over Core: Call CALC plugin
  Core->>Calculate1: calculateVersionableObjects(versionableInfos: VersionableInfo[])

  Note over Calculate1: resolve last version for each versionable
  Calculate1->>Calculate1: resolveLastVersion(...): Version

  Note over Calculate1: resolve changes (commits + deps commits)
  Calculate1->>Calculate1: resolveChanges(paths: string[], since: string): Changes

  Note over Calculate1: build versionable tree based on dep graph
  Calculate1->>Calculate1: resolveDependencies(versionableInfos: VersionableInfo[]): VersionableInfo & {deps: VersionableInfo[]}
  Calculate1->>Core: return Versionable object(s)

  Note over Core: 3. PROCESS VERSIONS
  Note over Core: Call PROCESS plugin(s)

  Core->>Process1: TBD by the plugin(s)

  Note over Core: 4. POST
  Note over Core: Call all POST plugins
  
  Core->>Post1: plugin_i.execute(...)
  Post1->>Post1: export_versions()  
  
  Core->>Post2: plugin_i.execute(...)
  Post2->>Post2: tag_repo()

  Note over Core: finalize

As you can see, we still have many open points, so it’s the right moment to design the tool properly :D. I think it’s going to be complicated to define everything by text here, that’s why I’d insist to have a call and discuss the general strategy. Of course we would be more than glad to then summarize the points and document everything in this thread. It’s just to make the conversation easier.

You have my contact, so please don’t hesitate to reach out and suggest a time. We’re also based in Europe, so that helps with time slots.

Cheers,

[tiagomab99]

Hi @yjaaidi I have a question regarding these updates.
Assuming the following dep graph
apps/a => libs/a/ui => libs/x => libs/y
In case a feature was commited to y would a be incremented a feature?
Cheers

[hinogi]

Support for adding build number from CI