Unable to upgrade schema in Tarantool 3 after bootstrapping from 2.11

[yngvar-antonsson]

Bug description

I tried to bootstrap 3.1 replica from 2.11 cluster (Cartridge-based) and got error Field ''[7][20]'' was not found in the tuple while trying to call box.schema.upgrade()

• OS: MacOS
• OS Version: Sonoma 14.2.1
• Architecture: amd64

Tarantool 3.1.0-0-g96f6d88

Steps to reproduce

1. Setup replicaset on 2.11 (I used Cartridge)

2. Run tarantool 3.1 in interactive mode:
   tarantool -i

3. Connect tarantool 3.1 to 2.11 replicaset:
   box.cfg{replication = {'admin@localhost:3302', 'admin@localhost:3303'}}
   In logs:
   Your schema version is 2.11.0 while Tarantool 3.1.0-0-g96f6d88 requires a more recent schema version. Please, consider using box.schema.upgrade().
   Everything is OK, replica is bootstrapped and replication is alright.

4. Current schema:
   box.space._schema:select({})

   ---
   - - ['cluster', 'd46f25fa-078c-4f56-b6eb-be85212e9a8f'] # <- same uuid as in 2.11 cluster
     - ['max_id', 512]
     - ['version', 2, 11, 0]
     - ['vshard_version', 0, 1, 16, 0]

5. Call upgrade:
   box.schema.upgrade()
   In logs:

   2024-05-02 16:44:56.945 [64285] main/104/interactive/box.upgrade I> drop field max_id in space _schema
   2024-05-02 16:44:56.946 [64285] main/104/interactive/box.upgrade I> set schema version to 2.11.1
   2024-05-02 16:44:56.946 [64285] main/104/interactive/box.upgrade I> update replicaset uuid key
   2024-05-02 16:44:56.946 [64285] main/104/interactive/box.upgrade I> add instance names to _cluster
   2024-05-02 16:44:56.946 [64285] main/104/interactive/box.upgrade I> set schema version to 3.0.0
   2024-05-02 16:44:56.946 [64285] main/104/interactive/box.upgrade I> add trigger to _func
   Error received:

   ---
   - error: Field ''[7][20]'' was not found in the tuple
   ...

6. Current schema:
   box.space._schema:select({})

   ---
   - - ['replicaset_uuid', 'd46f25fa-078c-4f56-b6eb-be85212e9a8f']
     - ['version', 3, 0, 0]
     - ['vshard_version', 0, 1, 16, 0]
   ...

Actual behavior

Error received, any following calls of schema.upgrade return the same error.

Expected behavior

Schema should be updated. The other way, user should receive a clearer error message with details how to fix upgrade manually.

[yngvar-antonsson]

Even simpler repro:

tarantool           
Tarantool Enterprise 2.11.0-entrypoint-55-ga74aeaf7c-r508
type 'help' for interactive help
tarantool> box.cfg{}
^D

Then

>>> tarantool -i
Tarantool 3.1.0-0-g96f6d88
type 'help' for interactive help
tarantool> box.cfg{}
tarantool> box.schema.upgrade()
---
- error: Field ''[7][20]'' was not found in the tuple
...

[drewdzzz]

I cannot reproduce it without cartridge.
Does cartridge alter _space or _func? Could you, please, provide _space:format() and _func:format() from your snapshot? (Or better yet, the whole snapshot).

[drewdzzz]

Something is wrong with your snapshot.

Here I built Tarantool from 2.11.0 tag (Tarantool 2.11.0-0-g247a9a4183):

tarantool> box.space._vfunc:format()
---
- [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner', 'type': 'unsigned'}, {'name': 'name',
    'type': 'string'}, {'name': 'setuid', 'type': 'unsigned'}, {'name': 'language',
    'type': 'string'}, {'name': 'body', 'type': 'string'}, {'name': 'routine_type',
    'type': 'string'}, {'name': 'param_list', 'type': 'array'}, {'name': 'returns',
    'type': 'string'}, {'name': 'aggregate', 'type': 'string'}, {'name': 'sql_data_access',
    'type': 'string'}, {'name': 'is_deterministic', 'type': 'boolean'}, {'name': 'is_sandboxed',
    'type': 'boolean'}, {'name': 'is_null_call', 'type': 'boolean'}, {'name': 'exports',
    'type': 'array'}, {'name': 'opts', 'type': 'map'}, {'name': 'comment', 'type': 'string'},
  {'name': 'created', 'type': 'string'}, {'name': 'last_altered', 'type': 'string'}]
...

Here is _vfunc:format() from your snap:

tarantool> box.space._vfunc:format()
---
- [{'name': 'id', 'type': 'unsigned'}, {'name': 'owner', 'type': 'unsigned'}, {'name': 'name',
    'type': 'string'}, {'name': 'setuid', 'type': 'unsigned'}]
...

In your smaller repro I see line Tarantool Enterprise 2.11.0-entrypoint-55-ga74aeaf7c-r508. Tags like x.y.z-entrypoint are not actual Tarantool releases - they are needed for internal purposes (it's something like starter line for x.y.z release AFAIU), you need tag 2.11.0 to get actual Tarantool 2.11.0.

What's about broken upgrade, I see that format of _vfunc is extended on upgrade to 2.10.4, so your schema is broken - _schema contains [2, 11, 0], but it's actually not (schema 2.11.0 must have _vfunc with extended format). I guess, you checked out to commit which had schema 2.11.0 but it was "work-in-progress" and before actual 2.11.0 Release we modified 2.10.4 schema - this update became part of 2.11.0 schema as well (that's how it works in Tarantool), but you missed this update and left with broken schema.

[drewdzzz]

The problem is broken schema in snapshot, upgrade from real Tarantool 2.11 work fine.