Firebolt Node.js SDK

Installation

This library is published in the NPM registry and can be installed using any compatible package manager.

npm install firebolt-sdk --save

# For Yarn, use the command below.
yarn add firebolt-sdk

Using the library

import { Firebolt } from 'firebolt-sdk'

const firebolt = Firebolt();

const connection = await firebolt.connect({
  auth: {
    username: process.env.FIREBOLT_USERNAME,
    password: process.env.FIREBOLT_PASSWORD,
  },
  database: process.env.FIREBOLT_DATABASE,
  engineName: process.env.FIREBOLT_ENGINE_NAME
});

const statement = await connection.execute("SELECT 1");

// fetch statement result
const { data, meta } = await statement.fetchResult();

// or stream result
const { data } = await statement.streamResult();

data.on("metadata", metadata => {
  console.log(metadata);
});

data.on("error", error => {
  console.log(error);
});

const rows = []

for await (const row of data) {
  rows.push(row);
}

console.log(rows)

Contents

• About
• Documentation
• Usage
  • Create connection
    • ConnectionOptions
    • AccessToken
    • Service account
    • engineName
  • Test connection
  • Engine URL
  • Execute query
    • ExecuteQueryOptions
    • parameters
    • Named parameters
    • QuerySettings
    • ResponseSettings
  • Fetch result
  • Stream result
  • Result hydration
  • Engine management
    • getById
    • getByName
    • Engine
      • start
      • stop
      • restart
  • Database management
    • getById
    • getByName
    • Database
• Resource Manager
• Recipes
  • Streaming results
  • Custom stream transformers

About

The Firebolt client for Node.js. firebolt-sdk provides common methods for quering Firebolt databases, fetching and streaming results, and engine management.

firebolt-sdk supports Node.js > v14.

Documentation

Usage

Create connection

const connection = await firebolt.connect(connectionOptions);

ConnectionOptions

type UsernamePasswordAuth = {
  username: string;
  password: string;
};

type AccessTokenAuth = {
  accessToken: string;
};

type ServiceAccountAuth = {
  client_id: string;
  client_secret: string;
};

type ConnectionOptions = {
  auth: UsernamePasswordAuth | AccessTokenAuth | ServiceAccountAuth;
  database: string;
  engineName?: string;
  engineEndpoint?: string;
  account?: string;
};

engineName

You can pass engineName: "system" to use system engine, which is always on and execute AQL queries on it. If engineName or engineEndpoint was not passed sdk will fallback to default engine endpoint for database.

AccessToken

Instead of passing username/password directly, you can also manage authentication outside of node sdk and pass accessToken when creating the connection

const connection = await firebolt.connect({
  auth: {
    accessToken: "access_token",
  },
  engineName: 'engine_name',
  account: 'account_name',
  database: 'database',
});

Service account

Instead of passing username/password, you can also use service account

const connection = await firebolt.connect({
  auth: {
    client_id: 'b1c4918c-e07e-4ab2-868b-9ae84f208d26';
    client_secret: 'secret';
  },
  engineName: 'engine_name',
  account: 'account_name',
  database: 'database',
});

Test connection

TODO: write motivation connection can be tested using:

const firebolt = Firebolt();
await firebolt.testConnection(connectionOptions)

which will perform authentication and simple select 1 query

Engine URL

Firebolt engine URLs use the following format:

<engine-name>.<account-name>.<region>.app.firebolt.io

For example: your-engine.your-account.us-east-1.app.firebolt.io. You can find and copy your engine endpoint name in the Firebolt web UI.

Execute Query

const statement = await connection.execute(query, executeQueryOptions);

Execute Query with set flags

const statement = await connection.execute(query, {
  settings: { query_id: 'hello' }
});

ExecuteQueryOptions

export type ExecuteQueryOptions = {
  parameters:? unknown[];
  settings?: QuerySettings;
  response?: ResponseSettings;
};

parameters

parameters field is used to specify replacements for ? symbol in the query.

For example:

const statement = await connection.execute("select ?, ?", {
  parameters: ["foo", 1]
});

will produce select 'foo', 1 query

Format Tuple:

import { Tuple } from 'firebolt-sdk'

const statement = await connection.execute("select ? where bar in ?", {
  parameters: [
    1,
    new Tuple(['foo'])
  ]
});

Named parameters

namedParameters field is used to specify replacements for :name tokens in the query.

For example:

const statement = await connection.execute("select :foo, :bar", {
  namedParameters: { foo: "foo", bar: 123 }
});

will produce select 'foo', 123 query

QuerySettings

Parameter	Required	Default	Description
output_format		JSON_COMPACT	Specifies format of selected data

You can also use QuerySettings to specify set flags. For example: { query_id: 'hello' }

ResponseSettings

Parameter	Required	Default	Description
normalizeData		false	Maps each row in response from array format to object
bigNumberAsString		false	Hydrate BigNumber as String

Fetch result

const { data, meta, statistics } = await statement.fetchResult();

The Promise API is not recommended for SELECT queries with large result sets (greater than 10,000 rows). This is because it parses results synchronously, so will block the JS thread/event loop and may lead to memory leaks due to peak GC loads.

It is recommended to use LIMIT in your queries when using the Promise API.

Stream result

const { data } = await statement.streamResult();
const rows: unknown[] = [];

data.on("metadata", metadata => {
  console.log(metadata);
});

data.on("error", error => {
  console.log(error);
});

for await (const row of data) {
  rows.push(row);
}

Result hydration

firebolt-sdk maps SQL data types to their corresponding JavaScript equivalents. The mapping is described in the table below:

Category	SQL type	JavaScript type	Notes
Numeric	INT	Number	If value cannot be represented by JavaScript Number (determine using Number.isSafeInteger), BigNumber from "bignumber.js" is used
	INTEGER	Number
	BIGINT	Number
	LONG	Number
	FLOAT	Number
	DOUBLE	Number
String	VARCHAR	String
	TEXT	String
	STRING	String
Date & Time	DATE	Date

Engine management

Engines can be managed by using the resourceManager object.

import { Firebolt } from 'firebolt-sdk'
const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const enginesService = firebolt.resourceManager.engine

getById

Returns engine using engine ID

import { Firebolt } from 'firebolt-sdk'
const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const engine = await firebolt.resourceManager.engine.getById(
  "c8a228ea-93df-4784-99f9-a99368518782",
);

getByName

Returns engine using engine name.

import { Firebolt } from 'firebolt-sdk'
const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const engine = await firebolt.resourceManager.engine.getByName("engine_name")

Engine

Property	Type	Notes
id	{engine_id: string; account_id: string}
name	string
endpoint	string
description	string
current_status_summary	string

Start

Starts an engine.

import { Firebolt } from 'firebolt-sdk'
const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const engine = await firebolt.resourceManager.engine.getByName("engine_name")
await engine.start()

Stop

Stops an engine.

import { Firebolt } from 'firebolt-sdk'
const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const engine = await firebolt.resourceManager.engine.getByName("engine_name")
await engine.stop()

Restart

Restarts an engine.

import { Firebolt } from 'firebolt-sdk'

const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const engine = await firebolt.resourceManager.engine.getByName("engine_name")
await engine.restart()

Database management

Databases can be managed by using the resourceManager object.

import { Firebolt } from 'firebolt-sdk'
const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const databaseService = firebolt.resourceManager.database

Database getById

Returns database using database ID

import { Firebolt } from 'firebolt-sdk'
const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const database = await firebolt.resourceManager.database.getById(
  "c8a228ea-93df-4784-99f9-a99368518782",
);

Database getByName

Returns database using database name.

import { Firebolt } from 'firebolt-sdk'
const firebolt = Firebolt();
await firebolt.connect(connectionOptions);
const database = await firebolt.resourceManager.database.getByName("database_name")

Database

Property	Type	Notes
id	{engine_id: string; account_id: string}
name	string
description	string

Resource Manager

It is possible to create resourceManager separately from firebolt client, providing only auth credentials

import { FireboltResourceManager } from 'firebolt-sdk'

const resourceManager = FireboltResourceManager();
await resourceManager.authenticate({
  auth: {
    username: process.env.FIREBOLT_USERNAME as string,
    password: process.env.FIREBOLT_PASSWORD as string,
  },
  account: process.env.ACCOUNT_NAME as string
});

const engine = await resourceManager.engine.getByName(
  process.env.FIREBOLT_ENGINE_NAME as string
);

Recipes

Streaming results

The recommended way to consume query results is by using streams.

For convenience, statement.streamResult also returns meta: Promise<Meta[]> and statistics: Promise<Statistics>, which are wrappers over data.on('metadata') and data.on('statistics').

const firebolt = Firebolt();

const connection = await firebolt.connect(connectionParams);

const statement = await connection.execute("SELECT 1");

const {
  data,
  meta: metaPromise,
  statistics: statisticsPromise
} = await statement.streamResult();

const rows: unknown[] = [];

const meta = await metaPromise;

for await (const row of data) {
  rows.push(row);
}

const statistics = await statisticsPromise

console.log(meta);
console.log(statistics);
console.log(rows)

Custom stream transformers

To achieve seamless stream pipes to fs or stdout, you can use the Transform stream.

import stream,  { TransformCallback } from 'stream';

class SerializeRowStream extends stream.Transform {
  public constructor() {
    super({
      objectMode: true,
      transform(
        row: any,
        encoding: BufferEncoding,
        callback: TransformCallback
      ) {
        const transformed = JSON.stringify(row);
        this.push(transformed);
        this.push('\n')
        callback();
      }
    });
  }
}

const serializedStream = new SerializeRowStream()

const firebolt = Firebolt();
const connection = await firebolt.connect(connectionParams);
const statement = await connection.execute("select 1 union all select 2");

const { data } = await statement.streamResult();

data.pipe(serializedStream).pipe(process.stdout);

Or use rowParser that returns strings or Buffer:

const { data } = await statement.streamResult({
  rowParser: (row: string) => `${row}\n`
});

data.pipe(process.stdout);

Development process

Actions before

Setup env variables

cp .env.example .env

Execute tests

  npm test

License

Released under Apache License.