Support for Postgres Pooling #363

starrematte · 2022-10-11T12:11:42Z

Added stable support for Pooling in Postgres.
A few changes have been made in order to achieve a consistent code.

deno.json and import_map.json have been added for support to some node libraries;
in the Connector interface, for code design issues I had to put _client and _connected as optional due to inchoerence with the pool existence;
PostgresPoolOptions has been added and relies on the original options of Postgres Poolclass;
PostgresConnector has been modified with substantial edits in the various exposed methods;
Database has been modified with the support of the underlying connector implementation of _pool with the method getPool?

eveningkid

Good evening, Matteo!

First off, thank you so much for contributing to this project–and for emailing me :)

I am usually swamped with work from all sides, it's hard for me to take time on open-source

Regarding this PR, looks solid. My comments are all to keep harmony in the codebase and should be easy to address

If not, feel free to raise any question/seek help in the comments and I'll be happy to assist :)

Have a great rest of your day!

eveningkid · 2022-10-14T15:01:23Z

import_map.json

+{
+  "imports": {
+    "https://dev.jspm.io/npm:@jspm/core@1/nodelibs/timers.js": "https://deno.land/std@0.159.0/node/timers.ts",
+    "https://dev.jspm.io/npm:@jspm/core@1/nodelibs/url.js": "https://deno.land/std@0.159.0/node/url.ts",
+    "https://dev.jspm.io/npm:@jspm/core@1/nodelibs/events.js": "https://deno.land/std@0.159.0/node/events.ts"
+  }
+}


I am sure this is a larger problem, but do you mind sharing what was causing this issue?

What parent lib?

Yes, i saw that some of these libraries relies on "dev.jspm.io" and are no longer available. But anyway, the parent lib is the main postgres lib. The error that gives is the following:

Module not found : "https://dev.jspm.io/npm:@jspm/core@1/nodelibs/url.js" at https://dev.jspm.io/pg-connection-string@2.5.0:3:8

one of the borken import for me is

denodb/deps.ts

Line 5 in 09eb699

export { default as SQLQueryBuilder } from "https://raw.githubusercontent.com/Zhomart/dex/930253915093e1e08d48ec0409b4aee800d8bd0c/mod-dyn.ts";

we can fix it by pointing to the branch instead of a commit.

https://github.com/Zhomart/dex/blob/update-deps/mod-dyn.ts

Yep sure, thanks!

lib/connectors/connector.ts

eveningkid · 2022-10-14T15:07:42Z

lib/connectors/connector.ts

+  _connected?: boolean | undefined;
+
+  /** Is the optional pool for making connections to an external instance. */
+  _pool?: ConnectionPool | undefined;


Same thing here, | undefined should be omitted :)

eveningkid · 2022-10-14T15:12:38Z

lib/connectors/connector.ts


  /** Test connection. */
  ping(): Promise<boolean>;

  /** Connect to an external database instance. */
-  _makeConnection(): void;
+  _makeConnection(): void | ConnectorClient | Promise<void> | Promise<ConnectorClient>;


We should keep this a void method

It makes a connection and shouldn't return anything

For the pool, we need it because of the underlying implementation. Should we keep it so?

eveningkid · 2022-10-14T15:25:59Z

lib/connectors/postgres-connector.ts

+    if (this._client) {
+      if (this._connected) {
+        return;
+      }
+
+      await this._client.connect();
+      this._connected = true;
+      return this._client;
+    } else {
+      return await this._pool!.connect();


Let's create two new methods:

_isPoolConnector() { return "size" in this._options; } _getClientOrPool() { return this._isPoolConnector() ? this.getPool() : this.getClient(); } _makeConnection() { if (this._connected) { return; } await this._getClientOrPool().connect(); this._connected = true; }

eveningkid · 2022-10-14T15:28:29Z

lib/connectors/postgres-connector.ts

  async ping() {
-    await this._makeConnection();
+    return await this.tryConnection(await this._makeConnection());
+  }

+  async tryConnection(client?: PostgresClient) {
    try {
      const [result] = (
-        await this._client.queryObject("SELECT 1 + 1 as result")
-      ).rows;
+        await client!.queryArray("SELECT 1 + 1 as result")
+      ).rows[0];
      return result === 2;
    } catch {
      return false;


Could we keep all the code inside ping()?

Now we'll need to call this._getClientOrPool() every time we need this._client or ._pool

Yes, as we stick to the this._getClientOrPool() we can keep everything inside ping()

eveningkid · 2022-10-14T15:29:10Z

lib/connectors/postgres-connector.ts

+    if (this._client) {
+      if (!this._connected) {
+        return;
+      }
+      await this._client.end();
+    } else {
+      await this._pool?.end()!
    }


Same thing here, using this._getClientOrPool().end() would be great :)

eveningkid · 2022-10-14T15:29:29Z

lib/database.ts

@@ -189,6 +189,11 @@ export class Database {
    return this.getConnector()._client;
  }

+  /* Get the database pool if existent. */
+  getPool?() {


Can we drop the ? here?

Suggested change

getPool?() {

getPool() {

For this, actually is up to the implementation to give a "pooling" experience on the connector. Afaik, not every connectors have a pool implementation out of the box, but prove me wrong

You're right, no problem!

eveningkid · 2022-10-14T15:33:34Z

lib/connectors/postgres-connector.ts

+   *  @example 
+   * //Direct connection usage:
+   * const connection = new PostgresConnector({
+   *    host: '...',
+   *    username: 'user',
+   *    password: 'password',
+   *    database: 'airlines',
+   * });
+   * //Pool connection usage:
+   * const connection = new PostgresConnector({
+   *    connection_params: {
+   *      host: '...',
+   *      username: 'user',
+   *      password: 'password',
+   *      database: 'airlines',
+   *    },
+   *    size: 5,
+   *    lazy: false
+   * });


I know this comes from a good intention but I believe types are good hints to use the connector

Types are always be up-to-date, we don't need to manually edit them each time

So let's keep your life easy here haha :D

Yeah you are right!

removed PostgresPool class as Connector is a base class Co-authored-by: Arnaud <eveningkid@users.noreply.github.com>

Co-authored-by: Arnaud <eveningkid@users.noreply.github.com>

starrematte · 2022-10-15T11:48:32Z

new commit: 3aeaec8

I fixed some stuff that I made and added yours. I did not test the other components involved with the Connector
changes.

If something seems strange, I forgot something or everything else, let's chat about it :)

* Added pool feature for the postgres connector * added stable support for postgres pool, added tests * Update lib/connectors/connector.ts * fixed stuff as previously mentioned in eveningkid#363 * making getClient & _getClientOrPool optional * update test Co-authored-by: Matteo Stellato <matteo.stellato.external@atos.net> Co-authored-by: Arnaud <eveningkid@users.noreply.github.com>

eveningkid

Another series of comments, thanks a lot for keeping up with them!

Also, please always run deno fmt at the root of your project before committing your files

It will keep all the syntax bits coherent with the rest of the codebase :)

lib/connectors/postgres-connector.ts

eveningkid · 2022-10-28T17:52:58Z

lib/connectors/postgres-connector.ts

  async _makeConnection() {
-    if (this._connected) {
-      return;
+    if (!this._isPoolConnector()) {
+      if (this._connected) {
+        return this._client!;
+      }
+      await this._client!.connect();
+      return this._client!;
+    } else if (this._pool?.available || !this._pool?.available) {
+      return await this.getPool()?.connect()
+    } else {
+      throw new Error("no connections available")
    }
+  }


Could makeConnection not return anything?

It was created to work as:

querySomething() { await makeConnection() // << make sure that this.client next line won't be undefined this.client.query(something) }

Only connect, don't return a client :)

eveningkid · 2022-10-28T17:55:32Z

lib/connectors/postgres-connector.ts

+      }
+      await this._client!.connect();
+      return this._client!;
+    } else if (this._pool?.available || !this._pool?.available) {


Not sure if you intended to write this?

if (a || !a)

I think this whole method could be rewritten as:

async _makeConnection() { if (this._connected) { return; } if (this._isPoolConnector()) { await this._getPool().connect(); } else { await this._getClient().connect(); } }

The problem here, like the previous comment is that if you do this:

await this._getClient().connect() // -> connects the client

this connects the actual client already instanciated before.
While doing this:

await this._getPool().connect() //-> gives a client back on which connection should be established

So that's why I make it return a client.

If we write the code like so:

async _makeConnection() { if (this._connected) { return; } if (this._isPoolConnector()) { await this._getPool().connect(); } else { await this._getClient().connect(); } }

we're going to get nothing.

We need something that makes us doing async client operations, and maybe could be like having an array of clients:

_clients?: Array<PostgresClient>;

where the first element is always instanciated if the connector is in "client" mode, while if in "pool" mode, the clients field grows until the max pool size choosen in the options.
Of course this has a bit of impact on the general design of the connector itself.

Another way could be to build a completely custom Pool class that every connector implement on their own. So the idea already shown with the clients could be possible without completely twsting the main design.

In a last option we can stick with _makeConnection() returning the client.
I do not see any better solutions at the moment, but something possibly will get in my mind soon.

@eveningkid what do you think?

eveningkid · 2022-10-28T17:57:30Z

lib/connectors/postgres-connector.ts

-      const [result] = (
-        await this._client.queryObject("SELECT 1 + 1 as result")
-      ).rows;
+      const connection = await this._makeConnection();


So now we can keep this outside of the try, as it was before:

async ping() { await this._makeConnection() try { const [result] = await this._getClientOrPool().queryArray(...)

And also, keep no console.log in your changes, thanks! :)

Forgot those while testing out😄

eveningkid · 2022-10-28T17:58:26Z

lib/connectors/postgres-connector.ts

    const query = this._translator.translateToQuery(queryDescription);
-    const response = await this._client.queryObject(query);
+    const response = await client!.queryObject(query);


I assume that every call to client should now call this._getClientOrPool()

eveningkid · 2022-10-28T17:59:24Z

lib/connectors/postgres-connector.ts

+    if (this._client) {
+      if (!this._connected) {
+        return;
+      }
+      await this._getClientOrPool().end();
    }


Let's keep things simple here:

async close() { if (!this.connected) { return; } await this._getClientOrPool().end(); this._connected = false; }

eveningkid · 2022-10-28T17:59:44Z

lib/database.ts

@@ -189,6 +189,11 @@ export class Database {
    return this.getConnector()._client;
  }

+  /* Get the database pool if existent. */
+  getPool?() {


You're right, no problem!

eveningkid · 2022-10-28T18:01:02Z

lib/connectors/connector.ts

+  /** Gets the client connected to the database */
+  getClient(): any
+
+  /** Gets the pool connected to the database */
+  getPool?(): any


Suggested change

/** Gets the client connected to the database */

getClient(): any

/** Gets the pool connected to the database */

getPool?(): any

/** Gets the client connected to the database */

getClient?(): ConnectorClient;

/** Gets the pool connected to the database */

getPool?(): ConnectorPool;

eveningkid · 2022-10-28T18:02:01Z

lib/connectors/connector.ts


  /** Options to connect to an external instance. */
  _options: ConnectorOptions;

  /** Is the client connected to an external instance. */
-  _connected: boolean;
+  _connected?: boolean


This is always defined:

Suggested change

_connected?: boolean

_connected: boolean

eveningkid · 2022-10-28T18:04:08Z

lib/connectors/connector.ts

@@ -16,19 +22,31 @@ export interface Connector {
  _translator: Translator;

  /** Client that maintains an external database connection. */
-  _client: ConnectorClient;
+  _client?: ConnectorClient;


Won't this cause type errors in other connectors?

Since it wasn't optional before, I assume that any code like:

this.client.doSomething()

Would not work anymore, complaining that this.client could be not defined

We could keep _client as always defined in this file and override this interface just for Postgres

Yes, this will have an impact in other files, so we can override it only in Postgres.

Co-authored-by: Arnaud <eveningkid@users.noreply.github.com>

starrematte added 2 commits October 4, 2022 13:29

Added pool feature for the postgres connector

b4d9e4c

added stable support for postgres pool, added tests

432e7aa

eveningkid reviewed Oct 14, 2022

View reviewed changes

starrematte and others added 3 commits October 14, 2022 17:59

Update lib/connectors/connector.ts

66ae125

removed PostgresPool class as Connector is a base class Co-authored-by: Arnaud <eveningkid@users.noreply.github.com>

Update lib/connectors/connector.ts

050d1c3

Co-authored-by: Arnaud <eveningkid@users.noreply.github.com>

fixed stuff as previously mentioned in eveningkid#363

3aeaec8

Aiko-Suzuki mentioned this pull request Oct 23, 2022

Postgress pooling Aiko-Suzuki/denodb#2

Merged

eveningkid reviewed Oct 28, 2022

View reviewed changes

starrematte and others added 3 commits October 31, 2022 12:02

Update lib/connectors/postgres-connector.ts

f87120b

Co-authored-by: Arnaud <eveningkid@users.noreply.github.com>

Merge branch 'eveningkid:master' into master

3ac09be

Merge branch 'eveningkid:master' into master

bc36df5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Postgres Pooling #363

Support for Postgres Pooling #363

starrematte commented Oct 11, 2022 •

edited

eveningkid left a comment

eveningkid Oct 14, 2022

starrematte Oct 14, 2022

Aiko-Suzuki Oct 16, 2022 •

edited

starrematte Oct 31, 2022

eveningkid Oct 14, 2022

eveningkid Oct 14, 2022

starrematte Oct 14, 2022 •

edited

eveningkid Oct 14, 2022

eveningkid Oct 14, 2022

starrematte Oct 14, 2022 •

edited

eveningkid Oct 14, 2022

eveningkid Oct 14, 2022

starrematte Oct 14, 2022

eveningkid Oct 28, 2022

eveningkid Oct 14, 2022

starrematte Oct 14, 2022

starrematte commented Oct 15, 2022

eveningkid left a comment

eveningkid Oct 28, 2022

eveningkid Oct 28, 2022

starrematte Oct 31, 2022

eveningkid Oct 28, 2022

starrematte Oct 31, 2022

eveningkid Oct 28, 2022

eveningkid Oct 28, 2022

eveningkid Oct 28, 2022

eveningkid Oct 28, 2022

eveningkid Oct 28, 2022

eveningkid Oct 28, 2022

starrematte Oct 31, 2022

Support for Postgres Pooling #363

Are you sure you want to change the base?

Support for Postgres Pooling #363

Conversation

starrematte commented Oct 11, 2022 • edited

eveningkid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Aiko-Suzuki Oct 16, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

starrematte Oct 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

starrematte Oct 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

starrematte commented Oct 15, 2022

eveningkid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

starrematte commented Oct 11, 2022 •

edited

Aiko-Suzuki Oct 16, 2022 •

edited

starrematte Oct 14, 2022 •

edited

starrematte Oct 14, 2022 •

edited