Hemera

A Node.js microservices toolkit for the NATS messaging system

• Website: https://hemerajs.github.io/hemera-site/

📓 Getting Started

Hemera (/ˈhɛmərə/; Ancient Greek: Ἡμέρα [hɛːméra] "day") is a small wrapper around the NATS driver. NATS is a simple, fast and reliable solution for the internal communication of a distributed system. It chooses simplicity and reliability over guaranteed delivery. We want to provide a toolkit to develop micro services in an easy and powerful way. We use bloom filters to provide a pattern matching RPC style. You don't have to worry about the transport. NATS is powerful.

With Hemera you have the best of both worlds. Efficient pattern matching to have the most flexibility in defining your RPC's. It doesn't matter where your server or client lives. You can add the same add as many as you want on different hosts to ensure maximal availability. The only dependency you have is a single binary of 7MB. Mind your own business NATS do the rest for you:

The key features of NATS in combination with Hemera are:

• Lightweight: The Hemera core is small as possible and can be extended by extensions or plugins.
• Service Discovery: You don't need a service discovery all subscriptions are managed by NATS.
• Load Balancing: Requests are load balanced (random) by NATS mechanism of "queue groups".
• Packages: Providing reliable and modern plugins to the community.
• High performant: NATS is able to handle million of requests per second.
• Scalability: Filtering on the subject name enables services to divide work (perhaps with locality) e.g. topic:auth:germany. Queue group name allow load balancing of services.
• Fault tolerance: Auto-heals when new services are added. Configure cluster mode to be more reliable.
• Auto-pruning: NATS automatically handles a slow consumer and cut it off.
• Pattern driven: Define the signatures of your RPC's in JSON and use the flexibility of pattern-matching.
• PubSub: Hemera supports all features of NATS. This includes wildcards in subjects and normal publish and fanout mechanism.
• Tracing: Any distributed system need good tracing capabilities. We provide support for Zipkin a tracing system which manages both the collection and lookup of this data.
• Monitoring: Your NATS server can be monitored by cli or a dashboard.
• Payload validation: Create your own validator or use existing plugins for Joi and Parambulator.
• Serialization: Use JSON, Msgpack or Avro to serialize your data (dynamic or static).
• Metadata: Transfer metadata across services or attach contextual data to tracing systems.
• Dependencies: NATS is a single binary of 7MB and can be deployed in seconds.

Packages

The hemera repo is managed as a monorepo, composed of multiple npm packages.

General	Version
nats-hemera
hemera-zipkin
hemera-store

Messaging bridges	Version
hemera-rabbitmq
hemera-nsq

Database adapter	Version
hemera-arango-store
hemera-sql-store
hemera-elasticsearch
hemera-couchbase-store

Payload validation	Version
hemera-joi
hemera-parambulator

Data serialization	Version
hemera-msgpack
hemera-avro

Cache	Version
hemera-redis-cache

Table of contents

• Prerequisites
• Installing
• Example
• Writing an application
  • Define your service
  • Call your service
• Pattern matching rules
  • Matched!
  • Not matched!
• Error handling
  • Reply an error
  • Error-first-callbacks
  • Handle timeout errors
  • Fatal errors
  • Listen on transport errors
  • Specify custom timeout per act
• Delegation
  • Metadata
  • Context
  • Delegate
• Extension points
  • Client
  • Server
• Tracing capabilities
  • Life-cycle events
• Publish & Subscribe
• Payload validation
• Plugins
• Logging
• Protocol
• Best practice
  • Multiple instances of your service
  • Clustering
• Introduction to NATS
• NATS Limits & features
• Bridge to other messaging systems
• Monitoring
• Nginx integration for NATS
• Contributing
• Inspiration

Prerequisites

NATS messaging system

We use the Request Reply concept to realize this toolkit. Request Reply

Installing

Example

'use strict';

const Hemera = require('nats-hemera');
const nats = require('nats').connect(authUrl);
    
const hemera = new Hemera(nats, { logLevel: 'info' });

hemera.ready(() => {


  hemera.add({ topic: 'math', cmd: 'add' }, (req, cb) => {

    cb(null, req.a + req.b);
  });

  hemera.add({ topic: 'email', cmd: 'send' }, (req, cb) => {

    cb();
  })


  hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 2, timeout$: 5000 }, (err, resp) => {
    
    console.log('Result', resp);
  });

  //Without callback
  hemera.act({ topic: 'email', cmd: 'send', email: 'foobar@mail.com', msg: 'Hi' });

});

Writing an application

Add: Define you implementation.

Act: Start a request

Topic: The subject to subscribe. The smallest unit of Hemera. It's kind of namespace for your service. If you want to scale your service you have to create a second instance of your service. If you just want to scale a method you have to subscribe to a different subject like math:additions because any subscriber have to contain the full implementation of the service otherwise you can run into a PatternNotFound exception.

Define your service

hemera.add({ topic: 'math', cmd: 'add' }, (req, cb) => {
  cb(null, req.a + req.b);
});

Call your service

hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1 }, (err, resp) => {
console.log(resp); //2
});

Pattern matching rules

A match happens when all properties of added pattern matches with the one in the passed object.

Matched!

hemera.add({ topic: 'math', cmd: 'add' }, (req, cb) => {
  cb(resp.a + resp.b)
});
hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1 });

Not matched!

hemera.add({ topic: 'math', cmd: 'add', foo: 'bar' }, (req, cb) => {
  cb(req.a + req.b)
});
hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1 });

Error handling

Reply an error

hemera.add({ topic: 'math', cmd: 'add' }, (req, cb) => {
  cb(new CustomError('Invalid operation'));
});

Error-first-callbacks

hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1 }, (err, resp) => {
 err instanceOf CustomError // true
});

Handle timeout errors

NATS is fire and forget, reason for which a client times out could be many things:

• No one was connected at the time (service unavailable)
• Service is actually still processing the request (service takes too long)
• Service was processing the request but crashed (service error)

hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1 }, (err, resp) => {
 err instanceOf TimeoutError // true
});

Fatal errors

Fatal errors will crash your server. You should implement a gracefully shutdown and use a process watcher like PM2 to come back in a clear state. Optional you can disable this behavior by crashOnFatal: false

hemera.add({ topic: 'math', cmd: 'add' }, (resp, cb) => {
  throw new Error('Upps');
});
hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1 }, (err, resp) => {
  err instanceOf FatalError // true
});

Listen on transport errors

const hemera = new Hemera(nats, { logLevel: 'info' });
hemera.transport.on('error', ...)
hemera.transport.on('disconnect', ...)
hemera.transport.on('connect', ...)
//see NATS driver for more events

Specify custom timeout per act

hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1, timeout$: 5000 }, (err, resp) => {
});

Delegation

* Notice the use of this

Metadata

If you want to transfer metadata to a service you can use the meta$ property before sending. It will be passed in all nested act. E.g. you can add a JWT token as metadata to express if your action is legitimate. Data will be transfered!

hemera.add({ topic: 'math', cmd: 'add' }, function (req, cb) {
    
    //Access to metadata
    let meta = this.meta$
    
    cb(null, req.a + req.b);
});

Will set the metadata only for this act and all nested operations. Data will be transfered!

hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1, meta$: { a: 'test' } }, function (err, resp) {

   this.act({ topic: 'math', cmd: 'add', a: 1, b: 5 });
});

Will set the metadata on all act. Data will be transfered!

hemera.meta$.token = 'ABC1234'
hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1}, function (err, resp) {
    //or
   this.meta$.token = 'ABC1234';

   this.act({ topic: 'math', cmd: 'add', a: 1, b: 5 });
});

Context

If you want to set a context across all act you can use the context$ property. Data will not be transfered!

hemera.context$.a = 'foobar';
hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1 }, function (err, resp) {
   
   this.context$.a // 'foobar'
   
   this.act({ topic: 'math', cmd: 'add', a: 1, b: 5 }, function (err, resp) {
        
       this.context$.a // 'foobar'
   });
});

If you want to set a context only for this act and all nested act

hemera.act({ topic: 'math', cmd: 'add', a: 1, b: 1, context$: 1 }, function (err, resp) {
   //or
   this.act({ topic: 'math', cmd: 'add', a: 1, b: 5 }, function (err, resp) {
        
      this.context$ // 1
   });
});

Delegate

If you want to pass data only to the add you can use delegate$. This feature is used to transfer contextual data to tracing systems.

hemera.act({ topic: 'math', cmd: 'add', delegate$: { foo: 'bar' } })

hemera.add({
  topic: 'math',
  cmd: 'add',
}, function (req, cb) {

  cb()

})

hemera.add({
  topic: 'math',
  cmd: 'add',
}, function (req, cb) {

  //Visible in zipkin ui
  this.delegate$.query = 'SELECT FROM User;'

})

Extension points

You can extend custom behavior by extensions.

Client extensions

• onClientPreRequest,
• onClientPostRequest

i s the index of the handler

hemera.ext('<client-extension>', function(next, i) {
   
   let ctx = this
   next(<error>)
})

Server extensions

• onServerPreHandler,
• onServerPreRequest
• onServerPreResponse

hemera.ext('<server-extension>', function (req, res, next, prevValue, i) {
   
   next(<error>)
   //res.send(<payload>)
   //res.end(<payload>)
})

• next() will call the next extension handler on the stack.
• res.send(<error> or <payload>) will end the request-response cycle and send the data back to the callee but other extensions will be called.
• res.end(<error> or <payload>) will end the request-response cycle and send the data back to the callee but no other extensions will be called.
• req contains the current request. Its an object with two properties payload and error because decoding issues. Payload and error object can be manipulated.
• res contains the current response. Its an object with two properties payload and error. Payload and error object can be manipulated.
• prevValue contains the message from the previous extension which was passed by send(<value>)
• i represent the position of the handler in the stack.

Tracing capabilities

Tracing in the style of Google’s Dapper

In any act or add you can access the property this.request$ or this.trace$ to get information about your current or parent call.

    meta: {}
    trace: {
      "traceId": "CRCNVG28BUVOBUS7MDY067",
      "spanId": "CRCNVG28BUVOLJT4L6B2DW",
      "timestamp": 887381084442,
      "duration": 10851,
      "service": "math",
      "method": "a:1,b:20,cmd:add,topic:math"
    }
    request: {
      "id": "CRCNVG28BUVONL3P5L76AR",
      "timestamp": 887381084459,
      "duration": 10851,
      "pattern": "a:1,b:20,cmd:add,topic:math"
    }
    result: 50

Request/response life-cycle events

Events:

• onClientPreRequest,
• onClientPostRequest
• onServerPreHandler,
• onServerPreRequest
• onServerPreResponse

hemera.on('<event>', (ctx) => {
  console.log(ctx)
})

Times are represented in nanoseconds.

Publish & Subscribe

Normal (One-to-many)

  //Subscribe
  hemera.add({
    pubsub$: true,
    topic: 'math',
    cmd: 'add',
    a: {
      type$: 'number'
    }
  }, (req) => {

  })
  //Publish
  hemera.act({
    pubsub$: true,
    topic: 'math',
    cmd: 'add',
    a: {
      type$: 'number'
    }
  });

Special - one-to-one (but with queue group names)

  //Subscribe
  hemera.add({
    topic: 'math',
    cmd: 'add',
    a: {
      type$: 'number'
    }
  }, (req) => {

  })
  //Publish
  hemera.act({
    pubsub$: true,
    topic: 'math',
    cmd: 'add',
    a: {
      type$: 'number'
    }
  });

Payload validation

You can use different validators e.g Joi example

hemera.add({
    topic: 'math',
    cmd: 'add',
    a: {
      type$: 'number'
    }
  }, (req, cb) => {

    cb(null, {
      result: req.a + req.b
    });
  });

Handling

hemera.act({ topic: 'math', cmd: 'add', a: '1' }, function (err, resp) {
        
   err instanceOf PayloadValidationError //true
});

Plugins

let myPlugin = function (options) {

  let hemera = this;
  
  //Expose data which you can access globally with hemera.exposition.<pluginName>.<property>
  hemera.expose('magicNumber', 42)

  hemera.add({
    topic: 'math',
    cmd: 'add'
  }, (req, cb) => {

    cb(null, {
      result: req.a + req.b
    });
  });

};

hemera.use({ plugin: myPlugin, attributes: { name: 'myPlugin' }, options: { } })

Logging

Hemera used Pino logger by default but you can also use your own example

Your custom logger have to support following log levels.

['info', 'warn', 'debug', 'trace', 'error', 'fatal']

[2017-02-04T22:19:34.156Z] INFO (app/2056 on starptech): Connected!
[2017-02-04T22:19:34.160Z] INFO (app/2056 on starptech): ADD - ADDED
    topic: "math"
    cmd: "add"
[2017-02-04T22:19:34.163Z] INFO (app/2056 on starptech):
    outbound: {
      "id": "5ecdad999267f031109df50f653a7f46",
      "pattern": "a:1,b:2,cmd:add,topic:math"
    }
[2017-02-04T22:19:34.167Z] INFO (app/2056 on starptech):
    inbound: {
      "id": "5ecdad999267f031109df50f653a7f46",
      "duration": 0.003826,
      "pattern": "a:1,b:2,cmd:add,topic:math"
    }

Protocol

Format: JSON

message ErrorCause {
  string message = 1;
  string name = 2;
}

message RootCause {
  string message = 1;
  string name = 2;
}

enum RequestType {
  pubsub = 0;
  request = 1;
}

message Error {
  string message = 1;
  string name = 2;
  Pattern pattern = 3;
  ErrorCause cause = 4;
  RootCause rootCause = 5;
  string ownStack = 6;
}

message Request {
  string id = 1;
  string parentId = 2;
  int64 timestamp = 3;
  int32 duration = 4;
  RequestType type = 5;
}

message Trace {
  string traceId = 1;
  string spanId = 2;
  int64 timestamp = 3;
  string service = 4;
  string method = 5;
  int32 duration = 6;
}

message Pattern = Object;
message Result = Any;
message Delegate = Object;
message Meta = Object;

message Protocol {
    Trace trace = 1;
    Request request = 2;
    Result result = 3;
    Error error = 4;
    Meta meta = 5;
    Delegate delegate = 6;
}

Best practice

Think in small parts. A topic is like a service. You can define a service like auth which is responsible for authenticate users.

This service has actions like:

hemera.add({ topic: 'auth', cmd: 'authenticate' })
hemera.add({ topic: 'auth', cmd: 'passwordReset' })
...

Create multiple instances of your service.

Now your service is scaled.

node service.js
node service.js

Create another NATS Server and create a cluster.

Now your service is fault-tolerant.

var servers = ['nats://nats.io:4222', 'nats://nats.io:5222', 'nats://nats.io:6222'];
var nc = nats.connect({'servers': servers});
new Hemera(nc);

Introduction to NATS

https://www.youtube.com/watch?v=NfL0WO44pqc

NATS Limits & features

http://nats.io/documentation/faq/

Are you the only one who use NATS for microservice architectures?

The simplicity and focus of NATS enables it to deliver superior performance and stability with a lightweight footprint. It has the potential of becoming the de-facto transport for microservice architectures and event driven systems in this new era.

Asim Aslam, Creator of Micro

"I discovered NATS for its performance, and stayed for its simplicity. It’s been a wonderfully reliable layer that connects our microservice architecture at Pressly. The source is easily readable and approachable, and the future developments keep me excited!

Peter Kieltyka - CTO, Pressly

Running the tests

Set the path to the gnatsd before start testing.

npm run test

Monitoring

Easy and beautiful tool to monitor you app. hemera-board

Bridge to other messaging systems

If you need message delivery or another guarantee which NATS cannot provide you are able to plugin any other messaging system. E.g we provide plugins for RabbitMQ and NSQ.

We can always build stronger guarantees on top, but we can’t always remove them from below." Tyler Treat

Nginx integration for NATS

nginx-nats

Built With

• Bloomrun - A js pattern matcher based on bloom filters
• Node Nats Driver - Node.js client for NATS, the cloud native messaging system.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

• Dustin Deus - StarpTech

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details

Inspiration

Seneca - A microservices toolkit for Node.js.