Return of the "From" command - Database state management

[redevill]

The following post and this discussion highlights the need to track DATABASE VERSION.
(Please refer to this post: https://medium.com/@smallbee/how-to-use-sequelize-sync-without-difficulties-4645a8d96841)

Consider

The first sequelize.sync() might deploy a version 1 (or x) database to a client. Dev happens, migrations are created, and models are updated. More client deployments with various versions. Now, the latest version can be achieved in 2 ways:

1. New Install and sync with latest model definitions.
2. An existing deployment (Version x) is upgraded with post version x migrations to get Version Y.

What is version X?? What migrations need executing? Not those prior to version X.
These pre Version X migrations are not in sequelizeMeta! - and should not be, as this Database version was installed as X.

Deprecated Solution Umzug V2 & From command:

• All migrations (pre and post version X) are in the same source folder named according to the version they achieve.
• The database has a table with an Install Version stamp telling us what is "Version X"
• Given a particular database version X and using the command:
  { from: 'MyLastVersionX-MigrationTask', to: 'MyLastVersionY-MigrationTask' }
• Now we can apply only the appropriate migrations to Database Version X (only once because of sequelizeMeta)

Now the database state = the install version + migrations in sequelizeMeta

Discussion

There are - of course many solutions to this problem, but I believe that they all revolve around knowing the state of your database. In umzug's case, we are missing an indication of Initial condition onto which migrations are applied. Perhaps a database version "marker" can be inserted into sequelizeMeta, such that "pre-Marker" migrations would not be applied? Keeping umzug agnostic of the initial condition handling such as the sequelize.sync()

Personally, I can manage the problem given the From command back again. But this is an issue (per referenced article) that I think would be really good to address.

How should we handle database state management? After all, isn't that what migrations are trying to achieve in the first place?

[mmkal]

Per the upgrading from 2.x instructions in the readme:

• umzug.up({ from: '...' }) and umzug.down({ from: '...' }) are no longer supported. To run migrations out-of-order (which is not generally recommended), you can explicitly use umzug.up({ migrations: ['...'] }) and umzug.down({ migrations: ['...'] }).

Does this cover your usecase? If not, and you'd still like from to return, could you detail what your case would look like with and without it (i.e. using the above suggestion) to inform what it should look like?

[redevill]

In a word - yes, the From (and to) had a place and were useful.
I have had to implement a separate augmentation (mentioned below) in order to make it work.

Use-case With the "FROM":
This simplifies maintaining the database "FROM" install state, to current state.

1. Historical install of the database is at Version 2 (Sequelize / umzug ) provided initial database creation at that time with V2.
2. 'Today' a migration to current V4 is needed. (Migration library contains ALL migrations from V1 to current)
3. Using the install version (V2) and the "FROM" command, we can easily run ONLY the needed migrations from the library.
4. NOTE V1 migrations would not be blocked by umzug registration table, because they were never relevant to a V2 install.

Use-case Without the "FROM"
In addition to the above steps - Separate migration augmentation is needed.

1. This code must interrogate the migration library content
2. Using the Database version, determine the V2+ set of migration script names
3. Order and compile the list into the expected migration array.

[mmkal]

Your 'Use-case Without the "FROM"' sounds ok to me? It should be very rare that you'll need to do something unusual like that. The idea of the breaking change was to make sure you still can do that (by listing migrations, explicitly selecting whichever you like, then explicitly running them) but you're no longer encouraged to. Starting migrations from a mid-point is dangerous enough, usually, that I think it's OK it involves extra steps.

Is there anything you are actually unable to do with the new API though? Or, something you do frequently that's excessively inconvenient?

[redevill]

• "sounds ok to me?".... "rare" ... "unusual"

The use case using the "FROM" is needed with migrations from the "mid-point" almost EVERY time for me.

(not sure you are understanding the use - there are very few installs of the database that are V1 - ALL others MUST migrate "FROM" the "midpoint" (or database baseline) at which the database was installed)

The steps I outlined in the "without From" have been automated in my application, because hand typing the scripts into an array for the migrate is not acceptable - it's too easy to make a mistake. This means I CAN'T USE THE umzug CLI.

[mmkal]

I see. What do you think about a baseline feature instead? Llike flyway: https://documentation.red-gate.com/fd/baseline-184127456.html

It sounds like - and apologies if I'm still not understanding this right, I may have skimmed over some key details - this might be good for you:

node your-migrator.js baseline --name v7.js
node your-migrator.js up

I think I like that better because it's explicitly telling the migrations table (or whatever storage you use) "I know better, this is the state that the database is in".

What do you think?

[redevill]

YES! - Exactly the concept.
Not sure I understand your implementation - we need to know scripts where: MigrationScriptVersions[] <= BaseLine Version.
Does v7.js contain this information? "node your-migrator.js baseline --name v7.js"

One idea:
If the script names(MigrationScriptVersions[] <= BaseLine Version) are inserted into "SequelizeMeta" WITHOUT Running them...
Effectively you are telling the umzug system, they have already run.