REEM

Author: Trishul Nagenalli, updates and current maintenance by Kris Hauser

About

REEM (Redis Extendable Efficient Middleware) is a centralized middleware package for communication across distributed systems (e.g., robots). It is designed to be a single-package solution for passing information anywhere in the system while emphasizing ease of use and efficiency.

To make it easy, we chose to model information as a nested data structure that closely resembles python dictionaries. To the user, working with a database feels like working with a python dictionary. Out of the box, REEM supports communicating all native python types and numpy arrays.

To make it fast, we used Redis (an in-memory key-value database) running ReJSON (enabling Redis to store JSON data) as a central information store. To get maximum performance, we give users the power to control exactly how information is passed between the local program and Redis by defining their own marshallers. REEM comes with builtin marshallers for Numpy arrays.

REEM currently offers two communication paradigms:

• get/set database
• publish-subscribe

Installation

REEM is highly portable and runs on Python 3.3+.

To install the python package (and its dependencies), run

pip install reem

You will also need to have access to a Redis server with RedisJSON enabled. See the setup tutorial for step-by-step instructions on installing and configuring a compatible local Redis server.

Tutorials and API documentation

See example.py or the docs on read the docs.

Version history

0.1.3:

• Updated to require redis-py 4.0.0+. API is still backwards-compatible with redis-py 3.5+ (which requires rejson).
• Added with syntax on key-value store accessors which allows you to make many updates to an object without bogging down the Redis server. Usage is with kvs['key'] as val: foo(val) which is equivalent to val=kvs['key'].read(); foo(val); kvs['key'] = val and often much faster than foo(kvs['key']) if foo performs lots of reads and writes.
• Useless RedisInterface.initialize() method removed.
• Accessor KeyValueStore.get(key) will not fail if the top-level key exists on the Redis server but hasn't been pulled in this client.
• Added new convenience structure TolerantKeyValueStore which will automatically create uninitialized subkeys, throttle frequent reads, and accept both .-separated strings and lists of indexes as keys.

0.1.2:

• bug fixes in pub/sub implementation.
• Added get(key,default_value) to subkey accessor analogous to dict's get method.

0.1.1:

• added get() and set() to KeyValueStore for direct access to Rejson's JSON.GET / JSON.SET.

0.1.0: fork by Kris Hauser

• Can now access items by array index.
• Much easier to work with items like normal Python objects. Can:
  • Can treat accesssors as variables using read() and write(), e.g., var = server['key']; var.read(); var.write(x). (See bug fix note below).
  • Delete items via del server['key'] or del server['key']['subkey'] (uses Rejson's JSON.DEL)
  • Increment/decrement values via server['key']['subkey'] += 1 or -= (uses Rejson's JSON.NUMINCRBY). Note: does not work for Numpy arrays.
  • Multiply/divide values via server['key']['subkey'] *= 2 or /= (uses Rejson's JSON.NUMMULTBY). Note: does not work for Numpy arrays.
  • Append to arrays via server['key'].append(x) or server['key']['subkey'].append(x) (uses Rejson's JSON.ARRAPPEND)
  • Multiple append to arrays via server['key'] += [x,y,z] or server['key']['subkey'] += [x,y,z] (uses Rejson's JSON.ARRAPPEND)
  • Get sizes of arrays/objects using len(server['key']) or len(server['key']['subkey']) (uses Rejson's JSON.OBJLEN or JSON.ARRLEN)
  • Get types of items using server['key'].type() or server['key']['subkey'].type() (uses Rejson's JSON.TYPE)
• KeyValueStore, PublishSpace, SilentSubscriber, and CallbackSubscriber are now in the global reem namespace. Also, they can be given a string host rather than separately having to create a RedisInterface object.
• KeyValueStore and PublishSpace are now thread safe. (Note: not thoroughly tested yet).
• Objects retrieved by ['key'] no longer get clobbered when accessing ['subkey']. E.g.,

  kvs['topkey'] = {'subkey':{'foo':3}}
  a = kvs['topkey']
  b = a['subkey']
  b['foo'] = 4
  print(a.read())  #prior version unexpected prints {'subkey':{'foo':3},'foo':4}, new version prints {'subkey'{'foo':4}} as expected.
  

• Slight performance improvements for deeply nested accesses
• Python 2 version automatically returns json objects with keys / values as str instead of unicode.

0.0.x: original from Trishul Nagenalli