Adapting to the DMD monorepo

[kinke]

Hey guys!

I'm planning on significantly changing/simplifying the way we merge a new frontend incl. runtime libraries, now that upstream druntime has been merged into upstream dmd.

The rough idea is this:

1. Use the upstream dmd repo and add our ~1.3k druntime commits. Seems to be trivial: https://github.com/orgs/dlang/teams/team-dmd/discussions/1/comments/24
2. Rewrite the resulting repo via git filter-repo, via something like this:

# extract a subset of the dmd monorepo (druntime dir, dmd source dir, dmd tests dir and osmodel.mak)
git filter-repo --path druntime --path compiler/src/dmd --path compiler/test --path compiler/src/osmodel.mak
# remove unused files
git filter-repo --invert-paths \
--path 'compiler/src/dmd/backend' \
--path-regex 'compiler/src/dmd/scan(elf|mach|mscoff|omf)\.d' \
--path-regex 'compiler/src/dmd/lib(|elf|mach|mscoff|omf)\.d' \
--path 'compiler/src/dmd/link.d' \
--path 'compiler/src/dmd/e2ir.d' \
--path 'compiler/src/dmd/s2ir.d' \
--path-regex 'compiler/src/dmd/to(csym|ctype|cvdebug|dt|ir|obj)\.d' \
--path-regex 'compiler/src/dmd/iasm(|dmd|gcc)\.d' \
--path 'compiler/src/dmd/glue.d'
# move dirs/files
git filter-repo \
--path-rename compiler/src/dmd/:dmd/ \
--path-rename druntime/:runtime/druntime/ \
--path-rename compiler/test/:tests/d2/dmd-testsuite/ \
--path-rename compiler/src/osmodel.mak:tests/d2/src/osmodel.mak
# replace GitHub refs in commit messages (`#<number` => `dlang/dmd!<number>`)
git filter-repo --replace-message message-filters.txt
# TODO: only keep master/stable branches and tags

3. Push this as new ldc-developers/dmd-rewrite repo (or to ldc directly, with renamed branches + tags), and use a scheduled nightly CI job to keep it up-to-date.

Then one-time: merge our ldc repo with this (tag first-merged-dmd-druntime), right after merging #4141, and replacing the current druntime and dmd-testsuite submodules. Keep our frontend file versions with all LDC specifics, as well as the tests/d2/* files with our tweaks.

In the future, bumping frontend + runtime libraries should be as simple as merging the new tag (or stable branch) of the rewritten DMD repo, and bumping our Phobos submodule (still separate). So frontend, tests and druntime are bumped at once, and we'll have a git history for the frontend files too, with refs to the upstream PRs etc.

Any thoughts, concerns, previous experiences, tips… from your side? @JohanEngelen @thewilsonator @dnadlinger @redstar

[kinke]

Something to play with:

• test.sh:

#!/usr/bin/env bash
set -euo pipefail

rm -rf dmd-mono
git clone git@github.com:dlang/dmd.git dmd-mono
cd dmd-mono
git checkout first-merged-dmd-druntime
git checkout -b ldc-monorepo-init
git subtree pull --prefix druntime git@github.com:ldc-developers/druntime.git ldc-pre-monorepo -m "Merge LDC druntime patches"

# extract a subset of the dmd monorepo (druntime source + tests + Makefiles, dmd source + tests + osmodel.mak)
git filter-repo --force \
  --path druntime/src --path druntime/test --path-glob 'druntime/*.mak' \
  --path compiler/src/dmd --path compiler/test --path compiler/src/osmodel.mak \
  --path src --path test # required to keep git history before upstream druntime-merge (and associated directory movals)
# remove unused files
git filter-repo --invert-paths \
  --path-regex '(compiler/)?src/dmd/backend' \
  --path-regex '(compiler/)?src/dmd/root/(env|response|strtold)\.d' \
  --path-regex '(compiler/)?src/dmd/astbase\.d' \
  --path-regex '(compiler/)?src/dmd/cpreprocess\.d' \
  --path-regex '(compiler/)?src/dmd/dinifile\.d' \
  --path-regex '(compiler/)?src/dmd/dmdparams\.d' \
  --path-regex '(compiler/)?src/dmd/dmsc\.d' \
  --path-regex '(compiler/)?src/dmd/eh\.d' \
  --path-regex '(compiler/)?src/dmd/frontend\.d' \
  --path-regex '(compiler/)?src/dmd/scan(elf|mach|mscoff|omf)\.d' \
  --path-regex '(compiler/)?src/dmd/lib(|elf|mach|mscoff|omf)\.d' \
  --path-regex '(compiler/)?src/dmd/link\.d' \
  --path-regex '(compiler/)?src/dmd/(e|s)2ir\.d' \
  --path-regex '(compiler/)?src/dmd/to(csym|ctype|cvdebug|dt|ir|obj)\.d' \
  --path-regex '(compiler/)?src/dmd/iasm(|dmd)\.d' \
  --path-regex '(compiler/)?src/dmd/(objc_)?glue\.d' \
  --path-glob 'src/*.mak' # remaining Makefiles after moving dmd into compiler/
git filter-repo \
  `# move dirs/files` \
  --path-rename compiler/src/dmd/:dmd/ \
  --path-rename druntime/:runtime/druntime/ \
  --path-rename compiler/test/:tests/d2/dmd-testsuite/ \
  --path-rename compiler/src/osmodel.mak:tests/d2/src/osmodel.mak \
  `# prefix tags` \
  --tag-rename :dmd-rewrite- \
  `# replace GitHub refs in commit messages` \
  --replace-message ../message-filters.txt

• message-filters.txt:

regex:(^|\s|\()#(\d{2,})==>\1dlang/dmd!\2

[kinke]

Updated now. The result seems workable IMO - git histories for DMD and druntime are kept (incl. our druntime history), and there are hardly any unused extra files. E.g., I kept dmd/frontend.h on purpose, as it might sometime get into a shape where it becomes usable for us.

The commit messages are replaced as expected too - but GitHub refs of our druntime patches are wrongly replaced of course, and well, the upstream druntime patches before the monorepo merge obviously too. I guess we'll just have to live with that. Note that I chose the dlang/dmd!<number> format (like GitLab MR refs) to prevent all the upstream PRs from getting a notification.

[kinke]

The 3 git filter-repo invocations (somewhat optimized in the last iteration) take a total of about 80 secs on my box.

[kinke]

This is the draft for the full script:

#!/usr/bin/env bash
set -euxo pipefail

refs_prefix="dmd-rewrite-"

rm -rf dmd-mono
git clone git@github.com:dlang/dmd.git dmd-mono
cd dmd-mono

# merge LDC druntime patches (tag: first-merged-dmd-druntime-with-ldc-druntime-patches)
git checkout -b ldc.tmp first-merged-dmd-druntime --no-track
git subtree pull --prefix druntime git@github.com:ldc-developers/druntime.git ldc-pre-monorepo -m "Merge LDC pre-monorepo druntime patches"
git tag first-merged-dmd-druntime-with-ldc-druntime-patches

# extract a subset of the dmd monorepo (druntime source + tests + Makefiles, dmd source + tests + osmodel.mak)
git filter-repo --force \
  --path druntime/src --path druntime/test --path-glob 'druntime/*.mak' \
  --path compiler/src/dmd --path compiler/test --path compiler/src/osmodel.mak \
  --path src --path test # required to keep git history before upstream druntime-merge (and associated directory movals)
# remove unused files
git filter-repo --invert-paths \
  --path-regex '(compiler/)?src/dmd/backend' \
  --path-regex '(compiler/)?src/dmd/root/(env|response|strtold)\.d' \
  --path-regex '(compiler/)?src/dmd/astbase\.d' \
  --path-regex '(compiler/)?src/dmd/cpreprocess\.d' \
  --path-regex '(compiler/)?src/dmd/dinifile\.d' \
  --path-regex '(compiler/)?src/dmd/dmdparams\.d' \
  --path-regex '(compiler/)?src/dmd/dmsc\.d' \
  --path-regex '(compiler/)?src/dmd/eh\.d' \
  --path-regex '(compiler/)?src/dmd/frontend\.d' \
  --path-regex '(compiler/)?src/dmd/scan(elf|mach|mscoff|omf)\.d' \
  --path-regex '(compiler/)?src/dmd/lib(|elf|mach|mscoff|omf)\.d' \
  --path-regex '(compiler/)?src/dmd/link\.d' \
  --path-regex '(compiler/)?src/dmd/(e|s)2ir\.d' \
  --path-regex '(compiler/)?src/dmd/to(csym|ctype|cvdebug|dt|ir|obj)\.d' \
  --path-regex '(compiler/)?src/dmd/iasm(|dmd)\.d' \
  --path-regex '(compiler/)?src/dmd/(objc_)?glue\.d' \
  --path-glob 'src/*.mak' # remaining Makefiles after moving dmd into compiler/
git filter-repo \
  `# move dirs/files` \
  --path-rename compiler/src/dmd/:dmd/ \
  --path-rename druntime/:runtime/druntime/ \
  --path-rename compiler/test/:tests/d2/dmd-testsuite/ \
  --path-rename compiler/src/osmodel.mak:tests/d2/src/osmodel.mak \
  `# prefix tags` \
  --tag-rename ":$refs_prefix" \
  `# replace GitHub refs in commit messages` \
  --replace-message ../message-filters.txt

# create prefixed master/stable branches
git branch --no-track "${refs_prefix}master" master
git branch --no-track "${refs_prefix}stable" stable

# push prefixed master/stable branches and tags
# DO NOT UNCOMMENT!!!
#git remote add ldc git@github.com:ldc-developers/ldc.git
rm -rf ../ldc-mono && cp -R ../ldc ../ldc-mono && git remote add ldc ../ldc-mono/.git
git push --tags ldc "${refs_prefix}master" "${refs_prefix}stable"

Then:

$ cd ldc-mono
$ git rev-list dmd-rewrite-first-merged-dmd-druntime-with-ldc-druntime-patches..dmd-rewrite-stable | wc -l
249 # number of patches to DMD stable since the druntime merge in July
$ git rev-list dmd-rewrite-stable..dmd-rewrite-first-merged-dmd-druntime-with-ldc-druntime-patches | wc -l
1251 # LDC druntime patches

The big one-time merge would consist in merging LDC master with tag dmd-rewrite-first-merged-dmd-druntime-with-ldc-druntime-patches then.

[kinke]

I've just tested the incremental variant - calling the script a 2nd time without recreating ldc-mono. And hit the expected problem that the [dmd-rewrite-]first-merged-dmd-druntime-with-ldc-druntime-patches tag hash isn't stable.

BUT that doesn't matter - the rewrite is apparently stable even when completely excluding the merge-LDC-druntime-patches steps for the 2nd run! :) - So that'd be a first-time thing only, invoked manually, not by the proposed CI job automating the dmd-rewrite-* branch/tag updates.

[kinke]

I'd also like to take the opportunity to move dmd-testsuite (as we'll be stuck with the rewritten directory structure):

• tests/d2/dmd-testsuite => tests/dmd
• tests/d2/src/osmodel.mak => tests/dmd/osmodel.mak

[kinke]

An argument against automating the dmd-rewrite-* branch/tag updates are newly added files which are of no interest to LDC. While they can be excluded on the first merge to LDC master, it's a bit of a pain to discard changes in later merges if the file sees further development later (e.g., astbase.d). It's probably nicer (and viable I assume) to add such files to the script, having them removed altogether via filter-repo.

As it stands, a manual update is doable in less than 2 mins on my box, compared to currently 5+ mins (not measured) for updating dmd-testsuite, so it'd still be much better.

[kinke]

Well, manually checking for new extra files in dmd-rewrite-master (and checking if they are of interest) before pushing to the LDC repo is also a hassle and takes time. Hmm...

[kinke]

Well, this can be handled via something like

git fetch ldc "${refs_prefix}master"
numFiles="$(git diff --name-only --diff-filter=A --relative=dmd "ldc/${refs_prefix}master..${refs_prefix}master" | wc -l)"
if [[ "$numFiles" != 0 ]]; then
  set +x
  echo "$numFiles newly added DMD source files in branch master:"
  echo "----------"
  git diff --name-only --diff-filter=A --relative=dmd "ldc/${refs_prefix}master..${refs_prefix}master"
  echo "----------"
  if [[ "${FORCE:-0}" == 1 ]]; then
    echo "Adding these files due to env variable FORCE=1."
  else
    echo "Aborting due to these newly added files."
    echo "Please check if they are of interest to LDC:"
    echo "* If they aren't, have them removed via filter-repo in this script, then re-run the script."
    echo "* If they are, re-run this script with env variable FORCE=1."
    exit 1
  fi
  set -x
fi

[JohanEngelen]

Hey @kinke , great that you are working on this (and all the other LDC stuff), sorry I don't have much time to contribute.

I don't know how many downstream LDC forks there are, I only know of Weka's fork. Don't worry about Weka's fork. Weka's druntime deltas are small, and I will probably just manually reapply the changes to the monorepo.

For the new DMD file problem. Probably new files are added rarely.
I think merge conflicts on deleted files (in LDC's repo) won't happen if we merge DMD upstream after deleting a file from LDC. What I mean is:

     DMD   -------O---O---------C-----O-------
                       \__________     \
                        \         \     \
                         \         \     \
LDC dmd rewrite ----------M1---X---M2-----M3-----

where O is just a commit, M is a merge commit, X is the commit where an unnecessary file is removed from LDC, C is where a change to that file done in DMD. I think M3 will not give a merge conflict?

[kinke]

I'd need to double-check, but I think I've had such merge conflicts multiple times in the past, whenever the deleted file is changed.

Yep, confirmed:

CONFLICT (modify/delete): hello.dummy deleted in HEAD and modified in master.  Version master of hello.dummy left in tree.

[JohanEngelen]

I'd need to double-check, but I think I've had such merge conflicts multiple times in the past, whenever the deleted file is changed.

Yep, confirmed:

CONFLICT (modify/delete): hello.dummy deleted in HEAD and modified in master.  Version master of hello.dummy left in tree.

Indeed :(
But: using git mergetool it asks whether to keep the deleted file or the modified file. I found this: https://stackoverflow.com/a/41364780

yes d | git mergetool -- dmd_source_directory

to automatically keep all files deleted that were deleted in our repo.

Would that work?

[kinke]

That sounds like a pretty bad hack to me. ;) - So are you worried about the case that we might remove a file during the rewrite that we need later?

[JohanEngelen]

Ah, I thought for fully automating the merge (e.g. for CI system) that it'd be nice to be able to resolve the merge conflict automatically.

[kinke]

Ah okay, just that. :) - Well I think we can start out manually, just like we currently do for dmd-testsuite, and try to automate it sometime later. The only real problem are those new extra files; I also think they are going to be pretty rare, so we could just let the CI job be red in that case, to be resolved manually by a dev. If that only happens once every 2 months or so, it shouldn't be a big deal, plus having no auto-rewriting for a week or so until someone finds the time to look at these extra files shouldn't be an issue either.

And if it turns out to be a hassle, we can always choose to keep all new files for future rewrites, resorting to manual removal on the first merge and dealing with future merge conflicts.