GitHub ES/AE Parallel Migrator

Description

The GitHub ES/AE Parallel Migrator is a tool that allows you to migrate multiple repositories from GitHub ES/AE to GitHub Enterprise Cloud in parallel. It is written in Typescript and uses the GitHub REST API to start and monitor migrations.

Usage

The tool is designed to be executed as either a node application via a cli or via docker. Environment variables or arguments can be passed into the tool as well.

Environment Variables

Variable	Description	Example	Default
GITHUB_REPOS	A comma separated list of repositories to migrate	company-org/webapp,company-org/dashboards
GITHUB_AUTH_TOKEN	The personal access token with admin:org privileges to the org/repos.	ghp_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
GITHUB_ENDPOINT	The url of the api endpoint of the GH ES/AE instance	https://ghe-test.net/api/v3
GITHUB_OUTDIR	The output directory of the migration archives. Can be a full or relative path to the repository.	/user/home/archives	./archives
GITHUB_PRODUCTION	Boolean to determine whether the migration is for production (locks the source repository).	false	false
HTTP_PROXY	Proxy url for http traffic used by the Azure SDK.	http://10.10.1.10:1180
HTTS_PROXY	Proxy url for https traffic used by the Azure SDK.	https://10.10.1.10:1180

CLI

Requirements

• Node 18.x+
• NPM 9.5.x+
• Python 3.9.x+

From the root of the repository,

npm ci --only=production
npm run build

After the build is complete, you can either,

1. Create a .env file in the root of the repository and populate the environment variables. Then run the following command:

node dist/main.js

2. Run the following command and pass in the environment variables as arguments:

node dist/main.js --repos "company-org/webapp,company-org/dashboards" --production false --authToken "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"
--endpoint "https://ghe-test.net/api/v3" --outdir "archives"

Docker

Requirements

• Docker 20.10.x+

From the root of the repository,

docker build -t gh-parallel-migrator .

After the build is complete, you can either,

1. Create a .env file in the root of the repository and populate the environment variables. Then run the following command:

docker run --env-file .env --volume /user/home/dir:/app/archives gh-parallel-migrator

2. Run the following command and pass in the environment variables as arguments:

docker run --volume /user/home/dir:/app/archives gh-parallel-migrator --repos "company-org/webapp,company-org/dashboards" --authToken "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" --production true --endpoint "https://ghe-test.net/api/v3"

NOTE: The the second value (docker local dir) in the volume mount must match the correct output directory, or left out entirely and mapped to the default /app/archives directory.

Example: --volume /user/home/dir:/temp/archive ... --outdir /temp/archive/dir

Ouput

The tool will begin by making parallel REST requests to start migration exports on the GH ES/AE instance. Once the exports are complete, the tool will begin to monitor the status of the migrations. It will attempt to fetch the migration status every 10 seconds for 180 attempts (30 minutes). If the migration is not complete within the 30 minute window, the tool will exit with an error. If the migration is complete, the tool will download the migration archive and save it to the output directory.

A simplified example of the output is below:

Directory archives already exists
[
  { org: 'migration', migration_id: 128 },
  { org: 'migration', migration_id: 129 }
]
Migration 128 status: exporting.
Waiting 10 seconds before checking migration status again.
Migration 129 status: exporting.
Waiting 10 seconds before checking migration status again.
Migration 128 status: exported.
Migration 128 is complete.
Downloading migration 128 archive to archives/migration_archive_128.tar.gz.
Migration 129 status: exported.
Migration 129 is complete.
Downloading migration 129 archive to archives/migration_archive_129.tar.gz.
Migration 129 archive downloaded to archives/migration_archive_129.tar.gz.
Migration 128 archive downloaded to archives/migration_archive_128.tar.gz.
[
  { migrationId: 128, migrationStatus: 'exported' },
  { migrationId: 129, migrationStatus: 'exported' }
]

Contributing

The .devcontainer folder contains a devcontainer configuration for VSCode. This allows you to open the repository in a container and have all the necessary dependencies installed. It can also be use to run the tool with GitHub Codespaces.

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT