[POC] Migrating data with foreign key constraints #288
+424
−24
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| .. _copydbforeignkey: | ||
|
|
||
| =================================================================================== | ||
| Running ``ghostferry-copydb`` in production for tables with Foreign Key Constraints | ||
| =================================================================================== | ||
|
|
||
| Migrating tables with foreign keys constraints is an experimental feature in copydb and should be used at your own risk in production. | ||
|
|
||
|
|
||
| Prerequisites | ||
| ------------- | ||
|
|
||
| Before migrating tables with foreign key constraints via copydb there are a couple of things to take care of | ||
|
|
||
| - Ghostferry needs to be ran with ``SkipForeignKeyConstraintsCheck = true``, which will disable ghostferry to check foreign key | ||
| constraints during initialization. | ||
|
|
||
| - Source DB should be ``read_only``. | ||
|
|
||
| - Need to disable foreign key constraint checks on target DB by passing the following config to target db | ||
|
|
||
| .. code-block:: json | ||
|
|
||
| "Params": { | ||
| "foreign_key_checks": "0" | ||
| } | ||
|
|
||
| - Even though foreign key constraint checks are disabled on target db, table and db creation must happen in a specific order (eg parent should be created | ||
| before child table). This creation order can be specified by passing ``TablesToBeCreatedFirst`` in the config, or else the table creation order will be | ||
| figured out by copydb. | ||
|
|
||
| Limitations | ||
| ------------- | ||
|
|
||
| - Currently migrating tables with foreign key constraints is only possible if the source database is in read_only mode. Since tables with foreign key constraints can have referential actions for a foreign key such as ``ON DELETE CASCADE``, ``ON UPDATE CASCADE``. Cascading deletes and updates in child tablees caused by foreign key constraints don't show up in binlogs because these referential actions are dealt internally by InnoDB. This issue descibes briefly why the source database should be read_only during the migration - https://github.com/Shopify/ghostferry/issues/289. | ||
|
|
||
| - ``Interrupt-Resume`` functionality can be used as long as source database is read_only also during the interrupt period | ||
|
|
||
| - ``Inline Verifier`` can be used as long as it is ensured that the source database is read_only (even during the interrupt period) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -56,9 +56,12 @@ func (t *FerryTestSuite) TestSourceDatabaseWithForeignKeyConstraintFailsInitiali | |
| ferry = testhelpers.NewTestFerry().Ferry | ||
| ferry.Config.SkipForeignKeyConstraintsCheck = true | ||
|
|
||
| _, err = t.Ferry.SourceDB.Exec("SET GLOBAL read_only = ON") | ||
|
|
||
| err = ferry.Initialize() | ||
| t.Require().Nil(err) | ||
|
|
||
| _, err = t.Ferry.SourceDB.Exec("SET GLOBAL read_only = OFF") | ||
| } | ||
|
|
||
| func TestFerryTestSuite(t *testing.T) { | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,6 +3,7 @@ package test | |
| import ( | ||
| "fmt" | ||
| "sort" | ||
| "strings" | ||
| "testing" | ||
|
|
||
| "github.com/Shopify/ghostferry" | ||
|
|
@@ -444,6 +445,56 @@ func (this *TableSchemaCacheTestSuite) TestTargetToSourceRewritesErrorsOnDuplica | |
| this.Require().Equal(err.Error(), "duplicate target to source rewrite detected") | ||
| } | ||
|
|
||
| func (this *TableSchemaCacheTestSuite) TestGetTableCreationOrderWithoutForeignKeyConstraints() { | ||
| tables, err := ghostferry.LoadTables(this.Ferry.SourceDB, this.Ferry.TableFilter, nil, nil, nil, nil) | ||
| this.Require().Nil(err) | ||
|
|
||
| creationOrder, err := tables.GetTableCreationOrder(this.Ferry.SourceDB) | ||
| this.Require().Nil(err) | ||
|
|
||
| this.Require().Equal(len(creationOrder), 3) | ||
| this.Require().ElementsMatch(creationOrder, tables.AllTableNames()) | ||
| } | ||
|
|
||
| func (this *TableSchemaCacheTestSuite) TestGetTableCreationOrderWithForeignKeyConstraints() { | ||
| _, err := this.Ferry.SourceDB.Exec(fmt.Sprintf("CREATE TABLE `%s`.`table1` (`id1` BIGINT, PRIMARY KEY (`id1`))", testhelpers.TestSchemaName)) | ||
| this.Require().Nil(err) | ||
| _, err = this.Ferry.SourceDB.Exec(fmt.Sprintf("CREATE TABLE `%s`.`table2` (`id2` BIGINT, PRIMARY KEY (`id2`), CONSTRAINT `fkc2` FOREIGN KEY (`id2`) REFERENCES `table1` (`id1`))", testhelpers.TestSchemaName)) | ||
| this.Require().Nil(err) | ||
| _, err = this.Ferry.SourceDB.Exec(fmt.Sprintf("CREATE TABLE `%s`.`table3` (`id3` BIGINT, PRIMARY KEY (`id3`), CONSTRAINT `fkc3_1` FOREIGN KEY (`id3`) REFERENCES `table1` (`id1`), CONSTRAINT `fkc3_2` FOREIGN KEY (`id3`) REFERENCES `table2` (`id2`))", testhelpers.TestSchemaName)) | ||
| this.Require().Nil(err) | ||
|
|
||
| tables, err := ghostferry.LoadTables(this.Ferry.SourceDB, this.Ferry.TableFilter, nil, nil, nil, nil) | ||
| this.Require().Nil(err) | ||
|
|
||
| creationOrder, err := tables.GetTableCreationOrder(this.Ferry.SourceDB) | ||
| this.Require().Nil(err) | ||
|
|
||
| // 3 tests from the base test setup plus 3 added above | ||
| this.Require().Equal(len(creationOrder), 6) | ||
| this.Require().ElementsMatch(creationOrder, tables.AllTableNames()) | ||
|
|
||
| // verify the order: all we care for is that table1 is created before | ||
| // table2, which is created before table3 | ||
| table1Index := -1 | ||
| table2Index := -1 | ||
| table3Index := -1 | ||
| for i, tableName := range creationOrder { | ||
| if strings.HasSuffix(tableName, ".table1") { | ||
| table1Index = i | ||
| } else if strings.HasSuffix(tableName, ".table2") { | ||
| table2Index = i | ||
| } else if strings.HasSuffix(tableName, ".table3") { | ||
| table3Index = i | ||
| } | ||
| } | ||
| this.Require().NotEqual(table1Index, -1) | ||
| this.Require().NotEqual(table2Index, -1) | ||
| this.Require().NotEqual(table3Index, -1) | ||
| this.Require().True(table1Index < table2Index) | ||
| this.Require().True(table2Index < table3Index) | ||
| } | ||
|
|
||
| func TestTableSchemaCache(t *testing.T) { | ||
| testhelpers.SetupTest() | ||
| suite.Run(t, &TableSchemaCacheTestSuite{GhostferryUnitTestSuite: &testhelpers.GhostferryUnitTestSuite{}}) | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.