Support for ActiveRecord schema format :sql in migrations

[virtualstaticvoid]

Enable support for the :sql schema format in Rails. See ActiveRecord::Base.schema_format for details on schema_format.

For example, this is useful for Postgres databases using hstore.

NOTE
This method currently does not work for multi-schema databases, since the schema names are usually embedded in the structure.sql file.

[bradrobertson]

Hey thanks for the PR.

My only issue is for people using the schema based strategy. I'd want the adapter to throw an appropriate exception if someone has configured Apartment to use schemas AND the sql format. And sorry I'll get to your other PR asap

[virtualstaticvoid]

Agreed, I'll add that logic so that an exception is raised in that instance.

[NielsKSchjoedt]

Hi, I'm currently using both :sql as the schema_format, since I have some custom function in postgresql I need, and together with this I have multiple "databases" using the postgres schema method and I'm having hstore in a separate "hstore" schema. Could any of you explain me why this is a unintended behavior, and what issue I "should" encounter from doing that? And maybe what I should do instead?

[bradrobertson]

I haven't actually tested this out myself, but my understanding was that the generated schema.sql file would have the postgresql schema name embedded into the table names when it's generated. So if you had a users table, the schema.sql might look like:

CREATE TABLE public.users ...

This is a problem because Apartment uses the schema to populate a new tenant, which means you actually just want to create users, in that tenant's postgresql schema, but putting public in front would only create (or recreate) the table in the public schema.

I've never personally tried the sql format though, can you verify what I'm saying above to be true or false?

[NielsKSchjoedt]

Well, here is a glimpse of how it looks/works, it set's the search_path here and there in the file, not appending the schema:

--
-- PostgreSQL database dump
--

SET statement_timeout = 0;
SET client_encoding = 'UTF8';
SET standard_conforming_strings = on;
SET check_function_bodies = false;
SET client_min_messages = warning;

--
-- Name: da; Type: SCHEMA; Schema: -; Owner: -
--

CREATE SCHEMA da;


--
-- Name: SCHEMA da; Type: COMMENT; Schema: -; Owner: -
--

COMMENT ON SCHEMA da IS 'standard public schema';


--
-- Name: hstore; Type: SCHEMA; Schema: -; Owner: -
--

CREATE SCHEMA hstore;
CREATE EXTENSION hstore SCHEMA hstore;

SET search_path = da, pg_catalog;

--
-- Name: _final_mode(anyarray); Type: FUNCTION; Schema: da; Owner: -
--

CREATE FUNCTION _final_mode(anyarray) RETURNS anyelement
    LANGUAGE sql IMMUTABLE
    AS $_$
      SELECT a
      FROM unnest($1) a
      GROUP BY 1
      ORDER BY COUNT(1) DESC, 1
      LIMIT 1;
    $_$;


--
-- Name: mode(anyelement); Type: AGGREGATE; Schema: da; Owner: -
--

CREATE AGGREGATE mode(anyelement) (
    SFUNC = array_append,
    STYPE = anyarray,
    INITCOND = '{}',
    FINALFUNC = _final_mode
);


SET default_tablespace = '';

SET default_with_oids = false;

--
-- Name: advert_candidate_collector_fails; Type: TABLE; Schema: da; Owner: -; Tablespace: 
--

CREATE TABLE advert_candidate_collector_fails (
    id integer NOT NULL,
    exception_message text,
);

[virtualstaticvoid]

@NielsKSchjoedt Did you create this file, or was it dumped when running migrations?

[virtualstaticvoid]

Rereading @NielsKSchjoedt comment, I guess it could be possible to parse the schema search path and insert the new apartments schema prior to creating the new database schema.

I don't think this is a full proof solution though, as subsequent migrations, when dumped out, would include the schemas of apartments, which would be loaded when adding additional apartments.

[NielsKSchjoedt]

It was created during migration (nothing is changed except for the fact that I manually enhanced it to add CREATE SCHEMA hstore; because I needed it)

[virtualstaticvoid]

I don't think I'm understanding your setup as your question seems contradictory:

... currently using both :sql as the schema_format ...

AFAIK, the schema formats are mutually exclusive, so if the format is set to :sql then the schema.rb is ignored.

... I have multiple "databases" using the postgres schema method ...

The apartment implementation can only be configured to use one method (database or schema). If you have multiple databases, each with identical schema's then this pull request will work for you.

.... Could you explain me why this is a unintended behavior...

In the multi-schema scenario, with a schema format of :sql the structure.sql will contain any objects in the public schema (most likely created by your migrations) and any objects created manually which may be in other schemas, therefore it cannot be used to create a new schema for a new apartment.

Hope this helps...

[NielsKSchjoedt]

Hmm okay, so just to clerify myself on my setup:
Yes I am using the structure.sql format. And yes I'm having my tenants in postgres schemas (not databases).
My current problem with switching to the schema.rb format is that I won't be able to dump some custom sql-functions, that I have created, which my application (in all tenants) rely on. Too bad for me :-(

[bradrobertson]

hey @virtualstaticvoid in order for me to accept this can you make a few changes? This gem doesn't have a hard dependency on Rails, so your change would break any other Rack based apps using Apartment.

What I'd like to see is this as a config option that one can pass into Apartment.configure (ie: schema_format). This can be given a default in the railtie which only gets included for people using Rails. That way anyone else can manually set whatever option they like and we don't increase the number of external dependencies.

Also if you can rebase from dev so it can be auto merged that'd be great.

[virtualstaticvoid]

Cool, I'll have a go at improving it.

[virtualstaticvoid]

Ok, I've rebased off the development branch, introduced a schema_format configuration which is defaulted to the Rails configuration in the Railtie and the Apartment#database_schema_file method branches according to the schema format, unless a database_schema_file has been defined.

As a side note, a default for database_schema_file can only be determined when the gem is included in a Rails application. Not sure what the default should be for a Rack application, so it currently yields a nil value.

[kyledecot]

I just posted an issue that relates to this. Has there been any progress on getting this implemented?

[bradrobertson]

hey sorry, this kind of fell by the wayside. I'll take a look at this PR hopefully soon and get it merged in.

[kyledecot]

That would be much appreciated! Keep up the good work

[virtualstaticvoid]

Hi @bradrobertson

I've rebased off of the latest development branch, and pushed to a new branch sql_migration_2 on my clone.

If you want, I can update this pull request, or you can cherry-pick the commits when you are ready.

[JonasNielsen]

@virtualstaticvoid I just took your changes (virtualstaticvoid/apartment@27431faaf5f50d8b37dc5b384c6083be6828f98a) for a test spin.

Am I right that it doesn't support creation of databases with Apartment::Database.create('schema_name')?

> Apartment::Database.create('schema_name')
   (8.4ms)  CREATE SCHEMA "schema_name"
/myrailsproject/db/schema.rb doesn't exist yet

Schema format is configured to :sql in application.rb: config.active_record.schema_format = :sql

How are you creating new schemas and setting up the test database?

[virtualstaticvoid]

Hi @JonasNielsen

Ok, so you create your Rails application and Postgres database as usual. Next specify the config.active_record.schema_format = :sql in your application.rb and run rake db:migrate. This will generate the db/structure.sql file. (You can delete the db/schema.rb file to avoid confusion).

EDIT:
Assuming you are not using the schemas configuration for apartment (I.e. config.use_postgres_schemas = false), when you execute Apartment::Database.create('<tenant>') the new database gets created and the structure.sql will be applied to it.

Then after adding migrations, you can apply them to each database by running rake apartment:migrate. (Note this command runs rake db:migrate as well.)

The test database is created when you run rake db:create.

This pull request adds support for the :sql format, so that structure.sql can be applied when creating and migrating each tenant database.

(BTW: the revision you referred to is the current HEAD on development of this repo).

[JonasNielsen]

Thanks @virtualstaticvoid. The git ref was obviously incorrect - I wasn't testing your changes :)

Your instructions are clear. The issue now is that we're on Postgres, which appears not to be supported: Using the :sql schema format is not supported when using Postgres schemas..

I'm guessing that's because Postgres dumps CREATE SCHEMA statements to structure.sql?

It would be great to support this also. Any ideas where to start?

[virtualstaticvoid]

Ah apologies, I forgot about that important fact. I've corrected my comment above.

[virtualstaticvoid]

@JonasNielsen So a possible solution, albeit inelegant, would be to parse the structure.sql file to remove any CREATE SCHEMA statements. This isn't a foolproof solution though, since there may be other statements which could cause issues. E.g. Possibly the installation of PG extensions, user-defined data types.

[JonasNielsen]

Thanks @virtualstaticvoid I actually just started hacking on that. I'll share the results in this issue.

[virtualstaticvoid]

Also, a further complication, is that Rails would need to be (monkey) patched, so that when migrations are executed, and it runs pg_dump to produce the structure.sql that it excludes any "tenant" schemas.

See activerecord/lib/active_record/tasks/postgresql_database_tasks.rb

[virtualstaticvoid]

Perhaps the solution is the generate a secondary structure.sql file for this purpose, instead of parsing the existing one. The db:migrate task could be enhanced to include this new task.

Looking at the pg_dump command help, it appears there are some switches which may help to produce the file for a single schema (--schema, --exclude-schema) and be able to exclude "shared" models (--exclude-table).

[kyledecot]

Any plans to proceed w/ this?

[jhubert]

Bah. This just cost me hours trying to figure out why tests wouldn't run after switching to the :sql schema.

I understand the complexity, but I'm curious how everyone handles custom SQL commands with new tenants? I have a migration that creates some materialized postgresql views and it doesn't work with schema.rb. Should I just call a command.execute after a tenant is created?

[mariokam]

Hey guys, so what is the recommended way of running custom SQL functions during migrations in multiple-schemas? The current method swap_schema_qualifier in the PostgresqlSchemaFromSqlAdapter is swaping the public schema to the specific schema during migrations using regex, but it does not keep integrity of relationships between the public schema table and the local table in the private schema.