Skip to content

metarhia/globalstorage

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

GlobalStorage

TravisCI NPM Version NPM Downloads/Month NPM Downloads

The Concept

This is a distributed DBMS for technological stack Metarhia and it is built with following assumptions:

  • GS is designed to be built-in DBMS, to work inside Impress Applications Server; it is needed to avoid or minimize interprocess communication to access DB;
  • GS is compatible with JSTP JavaScript Transfer Protocol, all data slould be stored, stansmitted, handled and placed in RAM in the same format;
  • All data structures can be reduced to array representation to redice size by removing key names, we can do that with the help of metadata schemas and we can dynamicaly build prototypes from schemas and assign them to arrays, so getters/setters will convert access by name (hash) to assess by position (array);
  • Maximum memory usage, read-ahead and lazy-write, minimizing data conversion;
  • Using metadata everywhere, special declarative format for subject domein representation including fields, relations, and indices so we can automatically build a storage scheme in the relational database, memory structures and structure for the database, different the GUI, API server, etc.
  • The same API for client-side runtime and server-side runtime:
    • server-side storage engine;
    • client-side storage engine (multiple implementations for different platforms including mobile, desktop and browser);
    • sharding for distributed storage of large data amounts, geo-distribution, save backup copies, access load balancing;
    • allows user to exchange data in P2P mode;
  • Syncronization between client and server in realtime (close to realtime) and in lazy mode; so applications can work in online and offline (with locally stored data); having bidirectional data sync and hieratchical versioning like git have;
  • Global data structures unification for applications working with Metarhia technological stack: GlobalStorage, Impress, JSTP and Console through moderated distributed metadata repositories;
  • Ability to work with non-unified data structures (custom schemas), specific to certain subject domain;
  • GlobalStorage provides DAC (data access layer) abstraction, it substitutes ORM but it does not necessarily make maping on the relational model (though RDBMS are also supported);
  • Data structures have global distributed identification system, so data can be inserted anywhere and will not bring ID conflicts;
  • Data reliability is provided by distributed storage facilities, so each data structure should have master server and multiple backup and cache servers; using those servers GlobalStorage supports addressing, versioning and branching.

Metamodel Definition Language

Using this domain specific language we will describe subject domain in declarative format. To build GUI, API, business-loguic, data structures dynamically in runtime. For example we can build JavaScript prototype and assign it to positional array to access fields by name, so arrays will work like objects.

Example:

{
  code: { type: 'string', primary: true },
  name: {
    caption: 'City',
    type: 'string',
    size: 32,
    nullable: false,
    index: { unique: false },
    master: { dataset: 'Cities', key: 'name' }
  },
  birth: 'Date',
  city: 'string',
  addresses: {
    type: { array: 'Address' }
  },
  gender: {
    type: 'char',
    lookup: { dictionary: { M: 'Male', F: 'Female' } }
  },
  age: function() {
    var difference = new Date() - this.birth;
    return Math.floor(difference / 31536000000);
  }
}

Data types:

  • Built-in JavaScript types: string, number, boolean, Date, etc.
  • Global Storage types: id, uid, tree, ip, etc.
  • RDBMS data types: char, int, real, text, money, time, date...
  • Links to other data structures in GlobalStorage
  • Links to other data structures in Application

JavaScript Query Language

JSQL is a query language for data structures manipulation. JSQL have syntax for: filter, projection, dataset join and set operations. We have a separate repository for examples and specification: metarhia/JSQL. Current Implementation can be found in lib/transformations.js.

Contributors

See github for full contributors list

API

gs(provider, options)

Create provider

gs.schemaConfig

class Cursor

Cursor.prototype.constructor(options)

Cursor.prototype.definition(schema, category)

Returns: <this>

Attach schema

Cursor.prototype.enableLogging(provider, ctx, args)

Cursor.prototype.copy()

Returns: <Cursor> new instance

Copy references to new dataset

Cursor.prototype.clone()

Returns: <Cursor> new instance

Clone all dataset objects

Cursor.prototype.enroll(jsql)

Returns: <this>

Apply JSQL commands to dataset

Cursor.prototype.empty()

Returns: <this>

Remove all instances from dataset

Cursor.prototype.from(arr)

Returns: <Cursor> new instance

Synchronous virtualization converts Array to Cursor

Cursor.prototype.map(fn)

Returns: <this>

Lazy map

fn - <Function>, map function

Cursor.prototype.projection(fields)

  • fields: <string[]>|<Object> projection metadata array of field names or object with structure: { toKey: [ fromKey, functions... ] }

Returns: <this>

Declarative lazy projection

Cursor.prototype.filter(fn)

Returns: <this>

Lazy functional filter

Cursor.prototype.select(query)

Returns: <Cursor> new instance

Declarative lazy filter

Cursor.prototype.distinct()

Returns: <this>

Lazy functional distinct filter

Cursor.prototype.sort(fn)

Returns: <this>

Lazy functional sort

Cursor.prototype.order(fields)

Returns: <this>

Declarative lazy ascending sort

Cursor.prototype.desc(fields)

Returns: <this>

Declarative lazy descending sort

Cursor.prototype.count([field])

  • field: <string> field to use for count, optional

Returns: <this>

Calculate count

Cursor.prototype.sum(field)

Returns: <this>

Calculate sum

Cursor.prototype.avg(field)

Returns: <this>

Calculate avg

Cursor.prototype.max(field)

Returns: <this>

Calculate max

Cursor.prototype.min(field)

Returns: <this>

Calculate min

Cursor.prototype.col()

Returns: <this>

Convert first column of dataset to Array

Cursor.prototype.row()

Returns: <this>

Return first row from dataset

Cursor.prototype.one()

Returns: <this>

Get single first record from dataset

Cursor.prototype.limit(count)

Returns: <this>

Get first n records from dataset

Cursor.prototype.offset(offset)

Returns: <this>

Offset into the dataset

Cursor.prototype.union(cursor)

Returns: <this>

Calculate union and put results to this Cursor instance

Cursor.prototype.intersection(cursor)

Returns: <this>

Calculate intersection and put results to this Cursor instance

Cursor.prototype.difference(cursor)

Returns: <this>

Calculate difference and put results to this Cursor instance

Cursor.prototype.complement(cursor)

Returns: <this>

Calculate complement and put results to this Cursor instance

Cursor.prototype.selectToMemory(query)

async Cursor.prototype.continue(data)

Returns: <Promise>

Continue computations via i.e. MemoryCursor or other cursor

to handle remaining operations unsupported by current cursor

async Cursor.prototype.fetch([permissionChecker])

Returns: <Promise>

Get results after applying consolidated jsql

class StorageProvider

Abstract Storage Provider

StorageProvider.prototype.constructor(options)

Create StorageProvider

async StorageProvider.prototype.open(options)

Returns: <Promise>

Open StorageProvider

async StorageProvider.prototype.close()

Returns: <Promise>

Close StorageProvider

async StorageProvider.prototype.setup(options)

Returns: <Promise>

Setup StorageProvider

StorageProvider.prototype.enableLogging(ctx)

StorageProvider.prototype.error(name, ...ctx)

  • name: <string> error name that must be equal to one of the values from the Action's Errors field
  • ctx: <Array>

Utility method to generate <ActionError> from inside the Action

async StorageProvider.prototype.takeId()

Returns: <Promise>

Generate globally unique id

async StorageProvider.prototype.get(id[, permissionChecker])

Returns: <Promise>

Get object from GlobalStorage

async StorageProvider.prototype.getDetails(category, id, fieldName[, permissionChecker])

Returns: <Promise>

Get details for many-to-many link from GlobalStorage

async StorageProvider.prototype.set(obj[, permissionChecker])

Returns: <Promise>

Set object in GlobalStorage

async StorageProvider.prototype.create(category, obj[, permissionChecker])

Returns: <Promise>

Create object in GlobalStorage

async StorageProvider.prototype.update(category, query, patch[, permissionChecker])

Returns: <Promise>

Update object in GlobalStorage

async StorageProvider.prototype.delete(category, query[, permissionChecker])

Returns: <Promise>

Delete object in GlobalStorage

async StorageProvider.prototype.linkDetails(category, field, fromId, toIds[, permissionChecker])

  • category: <string> category with field having the Many decorator
  • field: <string> field with the Many decorator
  • fromId: <Uint64> Id of the record in category specified in the first argument
  • toIds: <Uint64>|<Uint64[]> Id(s) of the record(s) in category specified in the Many decorator of the specified field
  • permissionChecker: <Function> optional
  • Returns: <Promise>

Returns: <Promise>

Link records with Many relation between them

async StorageProvider.prototype.unlinkDetails(category, field, fromId, toIds[, permissionChecker])

  • category: <string> category with field having the Many decorator
  • field: <string> field with the Many decorator
  • fromId: <Uint64> Id of the record in category specified in the first argument
  • toIds: <Uint64>|<Uint64[]> Id(s) of the record(s) in category specified in the Many decorator of the specified field
  • permissionChecker: <Function> optional
  • Returns: <Promise>

Returns: <Promise>

Unlink records with Many relation between them

StorageProvider.prototype.select(category, query[, permissionChecker])

Returns: <Cursor>

Select objects from GlobalStorage

async StorageProvider.prototype.execute(category, action, actionArgs[, permissionChecker])

Returns: <Promise>

Execute an action

StorageProvider.prototype.log(op, args, ctx, response)

StorageProvider.prototype.getSystemSuffix(id)

Returns: <Uint64>

Get system suffix for given id

StorageProvider.prototype.curSystem(id)

Returns: <boolean>

Check whether data with given id is stored on this system

StorageProvider.prototype.getServerSuffix(id)

Returns: <Uint64>

Get server suffix for given id

StorageProvider.prototype.curServer(id)

Returns: <boolean>

Check whether data with given id is stored on this server

StorageProvider.prototype.getLocalId(id)

Returns: <Uint64>

Get id without system and server suffix

StorageProvider.prototype.parseId(id)

Returns: <Object>

  • systemSuffix: <Uint64> system suffix for given id
  • serverSuffix: <Uint64> server suffix for given id
  • localId: <Uint64> id without system and server suffix

Parse id

async StorageProvider.prototype.listApplications([filtererByRoles])

Returns: <Promise>

List all available applications

async StorageProvider.prototype.listCategories([filtererByPermission])

Returns: <Promise>

List all available categories

async StorageProvider.prototype.listActions([filtererByPermission])

Returns: <Promise>

List all available actions

StorageProvider.prototype.createJstpApi()

async StorageProvider.prototype.getSchemaSources({ categoryList, actionLists, appList } = {})

async StorageProvider.prototype.getCategoryL10n(langTag, category)

async StorageProvider.prototype.getDomainsL10n(langTag)

async StorageProvider.prototype.getCommonL10n(langTag)

async StorageProvider.prototype.getFormL10n(langTag, category, form)

async StorageProvider.prototype.getActionL10n(langTag, category, action)

class FsProvider extends StorageProvider

FsProvider.prototype.constructor(options // { path } where path is database location)

FsProvider.prototype.readStat(callback)

FsProvider.prototype.open(options, callback)

FsProvider.prototype.writeStat(callback)

FsProvider.prototype.close(callback)

FsProvider.prototype.takeId(callback)

FsProvider.prototype.get(id, callback)

FsProvider.prototype.create(obj, callback)

FsProvider.prototype.update(obj, callback)

FsProvider.prototype.delete(id, callback)

FsProvider.prototype.select(query, options, callback)

class MemoryProvider extends StorageProvider

MemoryProvider.prototype.constructor(callback)

MemoryProvider.prototype.close(callback)

MemoryProvider.prototype.create(obj, callback)

class PostgresProvider extends StorageProvider

PostgresProvider.prototype.constructor()

Create PostgresProvider

async PostgresProvider.prototype.open(options)

Returns: <Promise>

Open PostgresProvider

async PostgresProvider.prototype.close()

Returns: <Promise>

Close PostgresProvider

async PostgresProvider.prototype.setup(options)

Returns: <Promise>

Setup StorageProvider

async PostgresProvider.prototype.takeId(client)

Returns: <Promise>

Generate globally unique id

async PostgresProvider.prototype.getCategoryById(id)

async PostgresProvider.prototype.beginTx([options])

Returns: <Promise>

Begin transaction, returns a Promise that resolves in an object containing

some of the methods of the current provider and also the methods commit(), rollback(), and release(). For more detailed description of the options see https://www.postgresql.org/docs/current/sql-set-transaction.html

async PostgresProvider.prototype.get(id[, permissionChecker])

Returns: <Promise>

Get object from GlobalStorage

async PostgresProvider.prototype.getDetails(category, id, fieldName[, permissionChecker])

Returns: <Promise>

Get details for many-to-many link from GlobalStorage

async PostgresProvider.prototype.set(obj[, permissionChecker])

Returns: <Promise>

Set object in GlobalStorage

async PostgresProvider.prototype.create(category, obj[, permissionChecker])

Returns: <Promise>

Create object in GlobalStorage

async PostgresProvider.prototype.update(category, query, patch[, permissionChecker])

Returns: <Promise>

Update object in GlobalStorage

async PostgresProvider.prototype.delete(category, query[, permissionChecker])

Returns: <Promise>

Delete object in GlobalStorage

async PostgresProvider.prototype.linkDetails(category, field, fromId, toIds[, permissionChecker])

  • category: <string> category with field having the Many decorator
  • field: <string> field with the Many decorator
  • fromId: <Uint64> Id of the record in category specified in the first argument
  • toIds: <Uint64>|<Uint64[]> Id(s) of the record(s) in category specified in the Many decorator of the specified field
  • permissionChecker: <Function> optional
  • Returns: <Promise>

Returns: <Promise>

Link records with Many relation between them

async PostgresProvider.prototype.unlinkDetails(category, field, fromId, toIds[, permissionChecker])

  • category: <string> category with field having the Many decorator
  • field: <string> field with the Many decorator
  • fromId: <Uint64> Id of the record in category specified in the first argument
  • toIds: <Uint64>|<Uint64[]> Id(s) of the record(s) in category specified in the Many decorator of the specified field
  • permissionChecker: <Function> optional
  • Returns: <Promise>

Returns: <Promise>

Unlink records with Many relation between them

PostgresProvider.prototype.select(category, query)

  • category: <string> category to select the records from
  • query: <Object> fields conditions

Returns: <Cursor>

Select objects from GlobalStorage

class RemoteProvider extends StorageProvider

RemoteProvider.prototype.constructor(options = {})

async RemoteProvider.prototype.open(options)

  • options: <Object> options for jstp connection
    • transport: <string> jstp transport name
    • connectionArgs: <Array> arguments to be passed to corresponding transport's connect method

Returns: <Promise>

Open RemoteProvider

async RemoteProvider.prototype.close()

Returns: <Promise>

Close RemoteProvider

async RemoteProvider.prototype.get(id)

  • id: <string> globally unique record id

Returns: <Promise>

Get record from GlobalStorage

async RemoteProvider.prototype.getDetails(category, id, fieldName)

  • category: <string> category to get details in
  • id: <string> object id
  • fieldName: <string> field with the Many decorator

Returns: <Promise>

Get details for many-to-many link from GlobalStorage

async RemoteProvider.prototype.set(record)

Returns: <Promise>

Set record in GlobalStorage

async RemoteProvider.prototype.create(category, record)

Returns: <Promise>

Create record in GlobalStorage

async RemoteProvider.prototype.update(category, query, patch)

  • category: <string> category of record
  • query: <Object> record, example: { Id }
  • patch: <Object> record, fields to update

Returns: <Promise>

Update record in GlobalStorage

async RemoteProvider.prototype.delete(category, query)

Returns: <Promise>

Delete record in GlobalStorage

async RemoteProvider.prototype.unlinkDetails(category, field, fromId, toIds)

  • category: <string> category with field having the Many decorator
  • field: <string> field with the Many decorator
  • fromId: <Uint64> Id of the record in category specified in the first argument
  • toIds: <Uint64>|<Uint64[]> Id(s) of the record(s) in category specified in the Many decorator of the specified field

Returns: <Promise>

Unlink records with Many relation between them

async RemoteProvider.prototype.linkDetails(category, field, fromId, toIds)

  • category: <string> category with field having the Many decorator
  • field: <string> field with the Many decorator
  • fromId: <Uint64> Id of the record in category specified in the first argument
  • toIds: <Uint64>|<Uint64[]> Id(s) of the record(s) in category specified in the Many decorator of the specified field

Returns: <Promise>

Link records with Many relation between them

RemoteProvider.prototype.select(category, query)

Returns: <Cursor> cursor

Select record from GlobalStorage

async RemoteProvider.prototype.execute(category, action, actionArgs)

Returns: <Promise>

Execute an action

async RemoteProvider.prototype.getSchemaSources()

async RemoteProvider.prototype.listCategories()

async RemoteProvider.prototype.listCategoriesPermissions()

Returns: <Promise>

List categories permission flags

async RemoteProvider.prototype.listActions()

async RemoteProvider.prototype.listApplications()

async RemoteProvider.prototype.getCategoryL10n(langTag, category)

async RemoteProvider.prototype.getDomainsL10n(langTag)

async RemoteProvider.prototype.getCommonL10n(langTag)

async RemoteProvider.prototype.getFormL10n(langTag, category, form)

async RemoteProvider.prototype.getActionL10n(langTag, category, action)