How to upgrade using metadata.json and cli-migrations-v2 ?

[stephane303]

I am trying to upgrade from v1.3.3 to v2.5.0, in v1.3.3 I am creating an Hasura image with my metadata.json

FROM hasura/graphql-engine:v1.3.3.cli-migrations
RUN mkdir /hasura-migrations
COPY migrations/metadata.json /hasura-migrations/.

and this is working fine.

But if I use hasura/graphql-engine:v2.5.0.cli-migrations-v2,

FROM hasura/graphql-engine:v2.5.0.cli-migrations-v2
RUN mkdir /hasura-metadata
COPY migrations/metadata.json /hasura-metadata/.

I get the following error when starting my container:

time="2022-04-20T17:17:35Z" level=fatal msg="error applying metadata \n{\n "path": "$.args",\n "error": "key \"tables\" not found",\n "code": "parse-failed"\n}"

Until now I was just using a single metadata.json that was exported manually from the web console, what am I missing ?

[ilia-cy]

I'm having the same issue when trying to upgrade from v1.3.3 to v2.28.2.
No matter which variations I tried to do in regards to metadata.json location and its permissions, I still receive this error.

When trying to run the image without any metadata it loads properly, and the same metadata.json file is imported successfully through the UI .

Can you please advise?

[stephane303]

I am using cli-migrations-v3 now.

From hasura-cli I use: hasura metadata export --project <your directory>

and then I import this directory like this:

`FROM graphql-engine:v2.28.1.cli-migrations-v3

RUN mkdir /hasura-metadata

COPY /hasura-metadata/`

[ilia-cy]

@stephane303 Do you mean that in your v1.3.3 version you ran the export command, and then copied the output to the directory that is used in the Dockerfile?

Thanks!

[stephane303]

You can still manually import/export the json file from the interface, so yes for the initial migration I do that.

[ilia-cy]

Sorry could you please share some more details?
I tried to do what you did but it didn't work.

I did not understand what you copied in your Dockerfile (in this line COPY /hasura-metadata/).
Because when I try to put the exported json file from the interface there I still receive the same error.

[stephane303]

I mean you have to stop using metadata.json, it only works manually with the web interface.

[ilia-cy]

As far as I understand the entire point of the cli-migrations images is to let you import migrations and metadata upon service load.
This is what I need for development and CI purposes.

How can that work if it is only possible to do via the web interface?
How can I load the metadata in a CI pipeline for example?

[stephane303]

Yes that is what I do too, but not using metadata.json any more.

1. migrate to 2.*
2. export the configuration hasura metadata export --project <your directory>
3. Automatically import with

FROM graphql-engine:v2.28.1.cli-migrations-v3
RUN mkdir /hasura-metadata
COPY metadata/ /hasura-metadata/

Reference: https://hasura.io/docs/latest/migrations-metadata-seeds/auto-apply-migrations/

[ilia-cy]

Unfortunately that still doesn't work.

This is what i've done:

hasura@0dfe61c6895d:/var/tmp$ hasura-cli metadata export --project /var/tmp/
INFO config v1 is deprecated from v1.4
FATA[0000] please upgrade your project to a newer version.
use hasura scripts update-project-v2 to upgrade your project to config v2

hasura@0dfe61c6895d:/var/tmp$ hasura-cli scripts update-project-v2
INFO Updated config to version 2

I then used this directory to copy it to the image, just as you described above, but the same error occurred:

{"timestamp":"2023-07-10T08:01:53.000+0000","level":"info","type":"startup","detail":{"kind":"migrations-apply","info":"applying metadata from /hasura-metadata"}}
time="2023-07-10T08:01:54Z" level=fatal msg="error applying metadata \n{\n  \"error\": \"key \\\"tables\\\" not found\",\n  \"path\": \"$.args.metadata\",\n  \"code\": \"parse-failed\"\n}"

hasura@a88094f54af7:/hasura-metadata$ ll
total 160
drwxr-xr-x 1 root root   4096 Jul 10 07:54 ./
drwxr-xr-x 1 root root   4096 Jul 10 08:03 ../
-rw-r--r-- 1 root root      0 Jul 10 07:54 actions.graphql
-rw-r--r-- 1 root root     86 Jul 10 07:54 actions.yaml
-rw-r--r-- 1 root root      3 Jul 10 07:54 allow_list.yaml
-rw-r--r-- 1 root root      3 Jul 10 07:54 cron_triggers.yaml
-rw-r--r-- 1 root root      3 Jul 10 07:54 functions.yaml
-rw-r--r-- 1 root root      3 Jul 10 07:54 network.yaml
-rw-r--r-- 1 root root      3 Jul 10 07:54 query_collections.yaml
-rw-r--r-- 1 root root      3 Jul 10 07:54 remote_schemas.yaml
-rw-r--r-- 1 root root 120984 Jul 10 07:54 tables.yaml
-rw-r--r-- 1 root root     11 Jul 10 07:54 version.yaml

Does your metadata directory look similar to this?

[ilia-cy]

OK, so the problem was the usage of v3 with v2 config.

The solution for me was:

1. Load the v2 related image without any migrations or metadata:

FROM hasura/graphql-engine:v2.28.2.cli-migrations-v2

USER hasura

2. The image will load and will use the current metadata (that was generated by the previous v1.3.3 image).
   docker exec to the image and export the metadata to its v2 version, and then export the metadata using the cli.

hasura@0dfe61c6895d:/var/tmp$ hasura-cli metadata export --project /var/tmp/
INFO config v1 is deprecated from v1.4
FATA[0000] please upgrade your project to a newer version.
use hasura scripts update-project-v2 to upgrade your project to config v2

hasura@0dfe61c6895d:/var/tmp$ hasura-cli scripts update-project-v2
INFO Updated config to version 2

Note: there is a stupid bug that you have to have a config.yaml file when you try to export the metadata, so you can just touch config.yaml to workaround this.

3. Copy the contents of the exported content from the docker image:
   docker cp 0dfe61c6895d:/var/tmp/metadata metadata_v2

4. Update the original Dockerfile to use copy the migrated metadata directory:

FROM hasura/graphql-engine:v2.28.2.cli-migrations-v2

RUN mkdir /hasura-metadata
COPY metadata_v2/ /hasura-metadata/

USER hasura