Cannot test migrations if there is a collation in the migrations

[jrobichaud]

I have a migration with a collation:

Ex:

# Generated by Django 4.1.5 on 2023-02-04 09:26

from django.contrib.postgres.operations import CreateCollation
from django.db import migrations


class Migration(migrations.Migration):

    initial = True

    dependencies = []

    operations = [
        CreateCollation(
            "french",
            provider="icu",
            locale="fr-x-icu",
        )
    ]

When running the migration tests I have this error:

Error
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.DuplicateObject: collation "french" already exists


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/django_test_migrations/contrib/unittest_case.py", line 37, in setUp
    self.old_state = self._migrator.apply_initial_migration(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_test_migrations/migrator.py", line 61, in apply_initial_migration
    return self._migrate(targets, plan=plan)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_test_migrations/migrator.py", line 84, in _migrate
    return self._executor.migrate(migration_targets, plan=plan)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/migrations/migration.py", line 130, in apply
    operation.database_forwards(
  File "/usr/local/lib/python3.11/site-packages/django/contrib/postgres/operations.py", line 223, in database_forwards
    self.create_collation(schema_editor)
  File "/usr/local/lib/python3.11/site-packages/django/contrib/postgres/operations.py", line 199, in create_collation
    schema_editor.execute(
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/schema.py", line 199, in execute
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.ProgrammingError: collation "french" already exists

[jrobichaud]

I found this workaround:

class TestMyMigration(MigratorTestCase):
    databases = ("default", "my_db",)
    database_name = "my_db"
    # ...
    def setUp(self) -> None:
        sql.drop_models_tables(self.database_name)
        sql.flush_django_migrations_table(self.database_name)
        with connections["my_db"].cursor() as cursor:
            cursor.execute("DROP COLLATION french")
        super().setUp()
        
    def test_my_migration(self):
        pass

[skarzi]

hi 👋

Creating a collation is tricky because it's not a data migration, it's just changing the DB configuration/metadata. Probably the best solution will be to add some hook (something like, migrator.apply_*_migration(..., run_before_apply=drop_collaction)), so the developers can define their own code that will be run before applying initial/target migration. What do you think about it @sobolevn and @jrobichaud?

[sobolevn]

Yes, seems like a good idea. Something like setup= and teardown=

[jrobichaud]

Sounds great!

[sobolevn]

@jrobichaud would you like to send a PR? :)

[jrobichaud]

I could give it a try.

I have trouble finding meaningful names.

For the TestCase it could work:

• setUpBeforeApply
• tearDownAfterApply

For the migrator:

• run_before_apply (its not clear its for forward migration)
• run_after_apply (its not clear its for reverse migration)

In theory someone could want to run 4 different things depending if its before/after apply in forward/reverse migration.

I only need before apply in forward migration.

Any thoughts about this?

[jrobichaud]

Is anything needed for the migrator at all considering the user call it himself directly? He could just add the just before the function.

I believe only the TestCase needs new functions.

[skarzi]

yes, that's true that when the developers are using the Migrator instance directly they can add custom code before/after the apply_*_migration(), but I think that for apply_initial_migration we will usually want to run "before apply" hook/code just after dropping tables and flushing Django migrations table - so exactly here, then adding arguments to apply_*_migration makes sense. We can add identical arguments to the apply_target_migration to make the interface more intuitive.

So I would add following arguments to apply_*_migration and pass it down to _migrate():

• before_hook: Callable[[PlanType], None](executed in_migrate` before doing migration)
• • after_hook: Callable[[PlanType], None](executed in_migrate` after doing migration)

and for the unittest test case just 1 method (optionally 2), taking plan as well (like you proposed):

• run_before_initial_migration

Optionally also run_after_initial_migration, but this can be replaced by adding the required code to prepare (of course assuming the migration plan is not important).