automated migrations

[a-ba]

This commit:

• moves the sql migration scripts into the database image
• adds a new database named migrations to track the list of applied migrations
• overrides the entrypoint of the database image to have the migrations applied on startup and controlled with the SHANOIR_MIGRATION environment variable
• renames the migrations scripts to have them orderd by a sequence number rather than a timestamp (because it is easier to edit, and they will have to be edited in case the migration is created in a long-lived branch)
• adds a script in the github workflow to ensure that all new migration names are after all existing migration in the release branch

Pending issues:

• SHANOIR_MIGRATION=auto automatically attempts to apply the migration scripts, which may interfere with the other containers (because the microservices will also attempt to update their DB on SHANOIR_MIGRATION=auto). Perhaps it would better to have the database container to do nothing on SHANOIR_MIGRATION=auto)

• the ordering of migrations has to be deterministic and we may have an issue when dealing with long-lived branches. If any release happens before the branch is merged, then the ordering may be broken. We should have a CI test to ensure that all migrations added since the master branch have a greater sequence number.

[jcomedouteau]

I have a little question:
Why don't we have a value for SHANOIR_MIGRATION (auto for example) that applies the scripts THEN starts ? For development purpose that would be quite simpler, so that we don't have to restart docker-compose twice.

Otherwise it works quite well, maybe put a default value for SHANOIR_MIGRATION=auto in variable.env ?

[jcomedouteau]

Set auto as default on in docker-compose/database/variable.env ?

[a-ba]

Why don't we have a value for SHANOIR_MIGRATION (auto for example) that applies the scripts THEN starts ? For development purpose that would be quite simpler, so that we don't have to restart docker-compose twice.

This would interfere with hibernate. auto is already used for setting ddl-auto=update

case "$SHANOIR_MIGRATION" in
	auto)
		env_setdefault	spring.datasource.initialize	false
		env_setdefault	spring.jpa.hibernate.ddl-auto	update

Perhaps we can rename the current behaviour of auto as dev and use auto for the behaviour you are suggesting.

Now the question is: how should we report errors. Should we abort the container at startup (fail fast) or just ignore the errors. (I would prefer the first one).

Otherwise it works quite well, maybe put a default value for SHANOIR_MIGRATION=auto in variable.env ?

it is already set to auto in .env

[jcomedouteau]

Perhaps we can rename the current behaviour of auto as dev and use auto for the behaviour you are suggesting.

Now the question is: how should we report errors. Should we abort the container at startup (fail fast) or just ignore the errors. (I would prefer the first one).

I agree with you, the database container deployment should fail, to prevent any further database-microservice errors

it is already set to auto in .env

Yep but it's not taken into account on my local environnement => If nothing is set in variable.env, it fails.

[michaelkain]

minor modif by Anthony, and then we accept

[a-ba]

done

[jcomedouteau]

@a-ba there is a conflict if you can resolve it ?
I tried to test it, but I had some problem with the mysqld starting (Timeout during MySQL init => No logs in the mysql logs, it may be linked to my available memory, I don't really know).
@Nunien could you try to test it ?

[jcomedouteau]

Hello Anthony @a-ba !
I tested it again and fount out what may be the problem:

The mysql server seems to start correctly, but the shanoir-entrypoint.sh does not seems to detect it:

2021-09-09T13:38:50.867184Z 0 [Note] Server socket created on IP: '::'.
2021-09-09T13:38:50.884017Z 0 [Note] InnoDB: Buffer pool(s) load completed at 210909 13:38:50
2021-09-09T13:38:50.931357Z 0 [Note] Event Scheduler: Loaded 0 events
2021-09-09T13:38:50.931993Z 0 [Note] mysqld: ready for connections.
Version: '5.7.34' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL)
[Shanoir Entrypoint] Waiting for server
[...]
[Shanoir Entrypoint] Waiting for server
[Shanoir Entrypoint] Timeout during MySQL init.

In shanoir-entrypoint.sh l.40 this test

  if [ -f /mysql-init-complete ] && $MYSQLADMIN ping ; then

is never true, and after a test:

-f /mysql-init-complete

is the blocking part. When I removed it, it worked well.

Maybe it is linked to my mysql version ?
MySQL Docker Image 5.7.34-1.2.2-server

Some people actually try to connect to the database in such loop in entrypoint.shto wait for the disponibility, or check the logs for the "ready for connections." (but I find it quite ugly).
https://github.com/docker-library/docs/tree/master/mysql#no-connections-until-mysql-init-completes
docker-library/mysql#81

Jean-Côme

[jcomedouteau]

Hello @a-ba,
Let's discuss this for the next weekly. If we resolve the conflicts we'll be able to merge it. This would simplify a lot the different merges.
Thanks,
Jean-Côme

[julien-louis]

@michaelkain there are conflicts I don't know how to resolve in entrypoint_common. Besides, I don't know exactly what to to with the new sql scripts.

[jcomedouteau]

New SQL scripts should be renamed and moved to match the naming Anthony used:
/docker-compose/database/db-changes/[microservice]/[index]_[description].sql

[a-ba]

In utils/entrypoint_common (there were conflicts) I am a bit confused about the purpose of the spring.jpa.generate-ddl lines added by 9dd9bef (they look useless to me).

https://springhow.com/spring-boot-database-initialization/#h-so-what-s-with-spring-jpa-generate-ddl says:

But bear in mind that the spring.jpa.generate-ddl will be ignored if you have a config for spring.jpa.hibernate.ddl-auto. This
is because the second config is more fine-grained and thus overriding the first one.