Add cockroach backend to ci #2680

mkacper · 2020-03-20T13:24:26Z

This PR addresses using an experimental CockroachDB backend.

Here is the summary of the research around using CockroachDB and YugabyteDB with MongooseIM.

CockroachDB

At the first glance Cockroach seems to be a bit more mature project than Yugabyte. I've simply encountered less serious bugs when working with Cockroach. However, on the other hand, Cockroach has (in my opinion which is based on just several hours of playing with the db) a bit worse compatibility with Postgres drivers than Yuga. Below is the list of steps I've taken to make MIM work with Cockroach:

Modify pg.sql schema:
1. ENUM type is not supported.
2. family word has special meaning.
3. JSON fields don't want to work with pubsub (as far as I understand JSON escaped by MIM is not accepted by JSON type in the db).
4. Fields that are used in ON CONFLICT clause need to be UNIQUE, see: https://github.com/esl/MongooseIM/pull/2680/files#diff-e0bb6223ac581f8935ad246096a537d5R371.
String escaping - in general string escaping differs between Postgres and Cockroach:

MIM with Postgres:

(mongooseim@localhost)1> mongoose_rdbms:sql_query(<<"localhost">>, <<"SELECT 1;">>).
{selected,[{<<"1">>}]}
(mongooseim@localhost)2> mongoose_rdbms:sql_query(<<"localhost">>, <<"SELECT '\';">>).
{selected,[{<<>>}]}
(mongooseim@localhost)3> mongoose_rdbms:sql_query(<<"localhost">>, <<"SELECT '\\';">>).
{error,"unterminated quoted string at or near \"'\\';\""}
(mongooseim@localhost)4> mongoose_rdbms:sql_query(<<"localhost">>, <<"SELECT '\\\\';">>).
{selected,[{<<"\\">>}]}
(mongooseim@localhost)5>

MIM with Cockroach:

(mongooseim@localhost)25> mongoose_rdbms:sql_query(<<"localhost">>, <<"SELECT 1;">>).
{selected,[{<<"1">>}]}
(mongooseim@localhost)26> mongoose_rdbms:sql_query(<<"localhost">>, <<"SELECT '\';">>).
{selected,[{<<>>}]}
(mongooseim@localhost)27> mongoose_rdbms:sql_query(<<"localhost">>, <<"SELECT '\\';">>).
{selected,[{<<"\\">>}]}
(mongooseim@localhost)28> mongoose_rdbms:sql_query(<<"localhost">>, <<"SELECT '\\\\';">>).
{selected,[{<<"\\\\">>}]}
(mongooseim@localhost)29>

I've created a PoC for escaping problem, see: https://github.com/esl/MongooseIM/pull/2680/files#diff-98c5e078be0cdd48c01b07ef75dceb89L37-R57. It was necessary, for example, to make one of pubsub test cases pass. But it also required changing the server code. IMO to solve that problem we should write a proper unescaping function for CockroachDB and change some of the server code so DB results that might need unescaping go through that function.

Please notice that it's possible that there are more problems out there but I have not found it yet. I've started with mam_SUITE. When it passed I just went through SUITEs that failed on CI. So far I've looked into rdbms_SUITE and pubsub_SUITE.

This PR also contains code that sets up CockroachDB in a docker container and provisiones it to be ready to work with MIM.

YugabyteDB

I found some critical bugs e.g. ORDER BY clause causing different length of results or some weird errors when using concurrent connections to the DB. I've created an issue on Yuga github re the former issue and their team responded almost immediately and the bug has been solved within a few days - chapeau bas Yougabyte team!

When it comes the latter bug, here is the error:

reason={case_clause,#{class => throw,reason => {aborted,#{reason => "Query error: Restart read required at: { read: { physical: 1584977855699547 } local_limit: { physical: 1584977855704746 } global_limit: <min> in_txn_limit: <max> serial_no: 0 }",sql_query => ["SELECT ","nidx, p_key, name, type, owners, options"," from pubsub_nodes WHERE p_key = ",[39,<<"pubsub.localhost">>,39]," AND name = ",[39,<<"princely_musings_yeWbz8Oh">>,39]]}},stacktrace => [{mongoose_rdbms,sql_query_t,2,[{file,"/Users/kacpermentel/ESL/other/yugabyte/MongooseIM/_build/mim1/lib/mongooseim/src/rdbms/mongoose_rdbms.erl"},{line,239}]},{mod_pubsub_db_rdbms,find_node_by_name,2,[{file,"/Users/kacpermentel/ESL/other/yugabyte/MongooseIM/_build/mim1/lib/mongooseim/src/pubsub/mod_pubsub_db_rdbms.erl"},{line,316}]},{timer,tc,3,[{file,"timer.erl"},{line,197}]},{mod_pubsub_db_backend,find_node_by_name,2,[{file,[]},{line,103}]},{nodetree_dag,create_node,6,[{file,"/Users/kacpermentel/ESL/other/yugabyte/MongooseIM/_build/mim1/lib/mongooseim/src/pubsub/nodetree_dag.erl"},{line,56}]},{mod_pubsub,create_node_authorized_transaction,6,[{file,"/Users/kacpermentel/ESL/other/yugabyte/MongooseIM/_build/mim1/lib/mongooseim/src/pubsub/mod_pubsub.erl"},{line,1995}]},{mod_pubsub_db,'-extra_debug_fun/1-fun-0-',1,[{file,"/U..."},...]},...]}}

It comes form MIM logs when running pubsub suite. Setting DB workers to 1 in MIM config solved the problem.

Very strong advantage of YugabyteDB is its compatibility with Postgres driver. The DB SQL API behaves exactly the same as original Postgres one (as far as I could test it).

mongoose-im · 2020-03-20T14:53:21Z

8029.1 / Erlang 22.0 / small_tests / 6386f82
Reports root / small

8029.2 / Erlang 22.0 / internal_mnesia / 6386f82
Reports root/ big
OK: 1392 / Failed: 0 / User-skipped: 155 / Auto-skipped: 0

8029.4 / Erlang 22.0 / mysql_redis / 6386f82
Reports root/ big
OK: 2622 / Failed: 0 / User-skipped: 205 / Auto-skipped: 0

8029.3 / Erlang 22.0 / odbc_mssql_mnesia / 6386f82
Reports root/ big
OK: 2627 / Failed: 0 / User-skipped: 200 / Auto-skipped: 0

8029.5 / Erlang 22.0 / riak_mnesia / 6386f82
Reports root/ big
OK: 1525 / Failed: 0 / User-skipped: 162 / Auto-skipped: 0

8029.7 / Erlang 22.0 / elasticsearch_and_cassandra_mnesia / 6386f82
Reports root/ big
OK: 328 / Failed: 0 / User-skipped: 28 / Auto-skipped: 0

8029.6 / Erlang 22.0 / ldap_mnesia / 6386f82
Reports root/ big
OK: 1357 / Failed: 0 / User-skipped: 190 / Auto-skipped: 0

8029.9 / Erlang 21.3 / pgsql_mnesia / 6386f82
Reports root/ big / small
OK: 2640 / Failed: 0 / User-skipped: 187 / Auto-skipped: 0

mongoose-im · 2020-03-20T15:38:25Z

8030.1 / Erlang 22.0 / small_tests / 4a23066
Reports root / small

8030.2 / Erlang 22.0 / internal_mnesia / 4a23066
Reports root/ big
OK: 1392 / Failed: 0 / User-skipped: 155 / Auto-skipped: 0

8030.4 / Erlang 22.0 / mysql_redis / 4a23066
Reports root/ big
OK: 2622 / Failed: 0 / User-skipped: 205 / Auto-skipped: 0

8030.3 / Erlang 22.0 / odbc_mssql_mnesia / 4a23066
Reports root/ big
OK: 2627 / Failed: 0 / User-skipped: 200 / Auto-skipped: 0

8030.5 / Erlang 22.0 / riak_mnesia / 4a23066
Reports root/ big
OK: 1525 / Failed: 0 / User-skipped: 162 / Auto-skipped: 0

8030.7 / Erlang 22.0 / elasticsearch_and_cassandra_mnesia / 4a23066
Reports root/ big
OK: 328 / Failed: 0 / User-skipped: 28 / Auto-skipped: 0

8030.6 / Erlang 22.0 / ldap_mnesia / 4a23066
Reports root/ big
OK: 1357 / Failed: 0 / User-skipped: 190 / Auto-skipped: 0

8030.9 / Erlang 21.3 / pgsql_mnesia / 4a23066
Reports root/ big / small
OK: 16 / Failed: 204 / User-skipped: 23 / Auto-skipped: 1317

service_mongoose_system_metrics_SUITE:init_per_suite

{'EXIT',{test_case_failed,"service_mongoose_system_metrics is not running"}}

Add cockroach backend to ci #2680

Are you sure you want to change the base?

Add cockroach backend to ci #2680

Conversation

mkacper commented Mar 20, 2020 • edited

CockroachDB

YugabyteDB

mongoose-im commented Mar 20, 2020 • edited

mongoose-im commented Mar 20, 2020 • edited

mongoose-im commented Mar 25, 2020 • edited

mkacper commented Mar 20, 2020 •

edited

mongoose-im commented Mar 20, 2020 •

edited

mongoose-im commented Mar 20, 2020 •

edited

mongoose-im commented Mar 25, 2020 •

edited