CosmosDb Migration Runner - should it be open source?

[josdeweger]

At a client we are currently working on a CosmosDb Migration Runner inspired by Fluent Migrator. The language is c#, the Target Framework netstandard2.0. Very early stages, not properly battle tested yet. The basic idea is that per collection / document type combination you can migrate documents up to a newer version and down to an older version, as well as renaming collections (which is basically creating a new collection, transferring all the documents, and deleting the old one, since renaming a collection doesn't really exist in CosmosDb). We might introduce transferring documents by document type between collections in the near future as well. All this is achieved by writing migrations as c# classes.

This library brings us the following benefits:

• no more writing scripts that contain a lot of plumbing and are hard to keep track of and run in a consistent manner (especially not across teams)
• a deterministic way to execute migrations across environments
• an easy way to roll back migrations to an older version if deemed necessary (e.g. when your 'up' migration ended up being wrong after all when you migrated on prod...)
• a way to automate testing of migrations using the CosmosDb Emulator (meaning we can run the migration tests as part of a CI pipeline)
• changes made in our document structure reside alongside the migrations in version control

As we are building this, we were wondering if this might be a useful library to open source. Would developers be interested in using this? Since it is very generic functionality, we see no downside in sharing it with the community. For us it's a great way to get feedback and contributions. Looking forward to your feedback!

[jzabroski]

@josdeweger Certainly. I find it hard to believe that writing migrations in CosmosDB is business proprietary.

Heck, I even explain to people some of my more "secret sauce" knowledge as a data engineering consultant/expert, and how I use FluentMigrator here: #1387

If you want to contribute, you'd have to update all the tests

[josdeweger]

Hi @jzabroski, thanks for your reply!

• "I find it hard to believe that writing migrations in CosmosDB is business proprietary" - I think we are on the same page here. There is nothing related to the specific business domain my customer is in, it gives them nothing in terms of competitive edge, hence the idea to open source it (to give back and receive feedback, and maybe a contribution now and then)
• The example you give is awesome! The reddit post contains some of the exact arguments of why we created something (very simple, very small as of now) for CosmosDb
• "If you want to contribute, you'd have to update all the tests" - are you implying adding CosmosDb to FluentMigrations? I was thinking of open sourcing a separate library but I'm open to the idea. I'm not sure there is a level of abstraction high enough to be a) useful and b) covering both the mechanics of a sql and nosql db, but I might be wrong there... Curious of your thoughts there?

[jzabroski]

I had the same skepticism about NoSQL, but I also thought: If EFCore can support code-first migrations for CosmosDB (maybe they don't?), then why shouldn't FluentMigrator?

The hard part about integrating the first NoSQL vendor is doing all the logic mappings.

I've tried to make systemizing the logic mappings easier over time, but there is still more work to do, e.g. #1063

[josdeweger]

EFCore doesn't support migrations for CosmosDb afaik, interesting thread on the topic: dotnet/efcore#11099

[jzabroski]

I see. I agree with @JohnGalt1717 - What NoSQL really needs is a nicer data migration API. I was planning to create a parallel module within FluentMigrator, similar to Python's dbt. It's a great idea, but I don't have the time to execute on it.

[josdeweger]

For now I've decided to put what I have up on a personal repo. I can't make the time investment to try and integrate this with FluentMigrator (yet).

[josdeweger]

and here it is! https://github.com/josdeweger/CosmosDb.Migrator