[3.0 Beta] Changes not applying to tables when using schemaName

[maneike]

Link to reproduction

No response

Describe the Bug

I encountered a bug using payload-3.0-demo, where upon initializing Payload, a custom schema in my database is not being created. I've implemented it like so:

  db: postgresAdapter({
    pool: {
      connectionString: process.env.DATABASE_URI || "",
    },
    schemaName: "payload",
  }),

Here is the error after running simple pnpm dev:

@app/admin:dev:  ⨯ Internal error: error: schema "payload" does not exist
@app/admin:dev:     at /node_modules/pg-pool/index.js:45:11
@app/admin:dev:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
@app/admin:dev:     at async DrizzleORMPgClient.query (/node_modules/drizzle-kit/payload.js:34498:21)
@app/admin:dev:     at async apply (/node_modules/drizzle-kit/payload.js:36648:9)
@app/admin:dev:     at async pushDevSchema (/node_modules/@payloadcms/db-postgres/dist/utilities/pushDevSchema.js:47:5)
@app/admin:dev:     at async Object.connect (/node_modules/@payloadcms/db-postgres/dist/connect.js:84:5)
@app/admin:dev:     at async BasePayload.init (/node_modules/payload/dist/index.js:212:13)
@app/admin:dev: digest: "1355805570"

So I've created a schema with the specified name in my database and tables were properly created within it. I have the impression that this is a workaround for the problem, as when modifying the collection (e.g., changing the slug name in pages collection), I've received a similar error.

slug: "title" -> slug: "page_title"

Compiled in 448ms (4012 modules)
 ⨯ Internal error: error: column "page_title" does not exist
    at eval (./node_modules/.pnpm/pg@8.11.3/node_modules/pg/lib/client.js:526:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async eval (./node_modules/.pnpm/drizzle-orm@0.29.4_@libsql+client@0.5.6_@types+react@18.2.74_pg@8.11.3_react@18.2.0/node_modules/drizzle-orm/node-postgres/session.js:59:22)
    at async find (./node_modules/.pnpm/@payloadcms+db-postgres@3.0.0-beta.2_@types+react@18.2.74_payload@3.0.0-beta.2_react@18.2.0/node_modules/@payloadcms/db-postgres/dist/find/findMany.js:141:21)
    at async findOperation (./node_modules/.pnpm/payload@3.0.0-beta.2_@swc+core@1.4.12_@swc+types@0.1.6/node_modules/payload/dist/collections/operations/find.js:104:22)
    at async ListView (./node_modules/.pnpm/@payloadcms+next@3.0.0-beta.2_@types+react@18.2.74_http-status@1.6.2_monaco-editor@0.47.0_nex_yfexsn5fqfva4wuhfmx6xps6u4/node_modules/@payloadcms/next/dist/views/List/index.js:77:22)
digest: "2491816213"

To Reproduce

1. Create a postgres database (I've used Supabase)
2. Clone https://github.com/payloadcms/payload-3.0-demo and set up config and environment variables accordingly
3. Add schemaName: "payload" to payload.config.ts
4. Run pnpm dev and go to localhost:3000/admin

Payload Version

3.0.0-beta.6

Adapters and Plugins

@payloadcms/db-postgres

[DanRibbens]

This might be related to #5473

I am investigating this.

[DanRibbens]

I think this problem is coming from drizzle-kit. I'm waiting on answers from their team.

[maneike]

@DanRibbens
I've read that creating anyTable in public schema would fix the Drizzle issue. While it didn't fix it, now it tries to push the tables into the schema, however it's actually pushing to public, even after specifying custom schema in both Payload and Drizzle configs. Then after going through all the db push prompts successfully:

+ users table will be created
+ payload_preferences table will be created
+ payload_preferences_rels table will be created
+ payload_migrations table will be created
--- all table conflicts resolved ---

I'm getting the same error:

 ⨯ Internal error: error: schema "payload" does not exist
    at /node_modules/pg-pool/index.js:45:11
    ...

These are the queries I'm passing to be executed in db.execute(import_drizzle_orm9.sql.raw(query)); and they're being logged just before execution:

// node_modules/drizzle-kit/payload.js

    DrizzleORMPgClient = class extends DrizzleDbClient {
      async query(query, values) {
        console.log("console.log(query):", query)
        const res = await this.db.execute(import_drizzle_orm9.sql.raw(query));
        return res.rows;
      }
      async run(query) {
        console.log("I'm not being logged out due to error. :(")
        const res = await this.db.execute(import_drizzle_orm9.sql.raw(query));
        return res.rows;
      }
    };

Output:

console.log(query): select count(*) as count from "anyTable"
console.log(query): CREATE TABLE IF NOT EXISTS "payload"."users" (
        "id" serial PRIMARY KEY NOT NULL,
        "updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL,
        "created_at" timestamp(3) with time zone DEFAULT now() NOT NULL,
        "email" varchar NOT NULL,
        "reset_password_token" varchar,
        "reset_password_expiration" timestamp(3) with time zone,
        "salt" varchar,
        "hash" varchar,
        "login_attempts" numeric,
        "lock_until" timestamp(3) with time zone
);
 ⨯ Internal error: error: schema "payload" does not exist
    at /node_modules/pg-pool/index.js:45:11
    ...

So it looks like Drizzle is pointing to public first and only then looking for other schemas and eventually not finding them.

[K-Mistele]

I think this problem is coming from drizzle-kit. I'm waiting on answers from their team.

Probably not, I have used drizzle and drizzle-kit with non-default schemas (not the public schema) in numerous other projects without payload. It is either a payload issue, or an issue with how payload is using drizzle.

[K-Mistele]

I can share a few more helpful pieces of information:

1. The migration that is generated does not include a CREATE SCHEMA statement.

Note that the migration file generated below was created with pnpm run payload migrate:create using the following adapter config, AND that I have truncated the create statements since there are a lot:

    db: postgresAdapter({
      pool: {
        connectionString: process.env.DATABASE_URI || '',
      },
      schemaName: 'template_payload_nfc_app'

    }),

import { MigrateUpArgs, MigrateDownArgs, sql } from '@payloadcms/db-postgres'

export async function up({ payload }: MigrateUpArgs): Promise<void> {
await payload.db.drizzle.execute(sql`

CREATE TABLE IF NOT EXISTS "template_payload_nfc_app"."nfc_tags" (
	"id" serial PRIMARY KEY NOT NULL,
	"xuid" varchar NOT NULL,
	"active" boolean NOT NULL,
	"tag_name" varchar,
	"updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL,
	"created_at" timestamp(3) with time zone DEFAULT now() NOT NULL
);

CREATE TABLE IF NOT EXISTS "template_payload_nfc_app"."nfc_tags_rels" (
	"id" serial PRIMARY KEY NOT NULL,
	"order" integer,
	"parent_id" integer NOT NULL,
	"path" varchar NOT NULL,
	"media_gallery_id" integer
);

CREATE TABLE IF NOT EXISTS "template_payload_nfc_app"."users" (
	"id" serial PRIMARY KEY NOT NULL,
	"admin" boolean,
	"name" varchar,
	"updated_at" timestamp(3) with time zone DEFAULT now() NOT NULL,
	"created_at" timestamp(3) with time zone DEFAULT now() NOT NULL,
	"email" varchar NOT NULL,
	"reset_password_token" varchar,
	"reset_password_expiration" timestamp(3) with time zone,
	"salt" varchar,
	"hash" varchar,
	"login_attempts" numeric,
	"lock_until" timestamp(3) with time zone
);
`);

};

export async function down({ payload }: MigrateDownArgs): Promise<void> {
await payload.db.drizzle.execute(sql`

DROP TABLE "template_payload_nfc_app"."nfc_tags";
DROP TABLE "template_payload_nfc_app"."nfc_tags_rels";
DROP TABLE "template_payload_nfc_app"."users";
DROP TABLE "template_payload_nfc_app"."media_gallery_media_items";
DROP TABLE "template_payload_nfc_app"."media_gallery";
DROP TABLE "template_payload_nfc_app"."media_gallery_rels";
DROP TABLE "template_payload_nfc_app"."uploads";
DROP TABLE "template_payload_nfc_app"."payload_preferences";
DROP TABLE "template_payload_nfc_app"."payload_preferences_rels";
DROP TABLE "template_payload_nfc_app"."payload_migrations";`);

};

2. The generated JSON file does not include entries in the schemas field

Once again, I have not pasted the entire file - just the end of it

{
  "id": "3af332c8-094a-443d-a20e-410d184c2bbb",
  "prevId": "00000000-0000-0000-0000-000000000000",
  "version": "5",
  "dialect": "pg",
  "tables": {...}, // tables are here, correctly. omitted for illustration & brevity
  "enums": {},
  "schemas": {},
  "_meta": {
    "schemas": {}, // there should be stuff here for a schema, but there is not.
    "tables": {},
    "columns": {}
  }
}