Skip to content

In a nutshell, EinsteinDB is a persistent indexing scheme based off of LSH-KVX that exploits the distinct merits of hash index and B+-Tree index to support range scan and avoids long NVM writes for maintaining consistency; thus improving on LSH’s performance guarantees for skewed data and adopts ordered-write consistency to ensure crash consiste…

YosiSF/EinsteinDB

Repository files navigation

Theory

EinsteinDB is a Relativistic Causal Consistent Key-Value Store with a Semantic-Knowledge CQRS/EventSourcing engine, a modern implementation of NewSQL, and a powerful Blockchain Smart Contract Platform.

NewSQL

Our content addressable approach deploys an IPFS-style content addressable infrastructure for peer-to-peer sharing of state information. This contrasts with existing decentralized systems which often deploy a single data structure that is replicated across all peers and updated by consensus. Instead, we propose to use an IPFS-style content addressable architecture where each peer stores its own data in the form of Merkle roots stored on filtrons (a new type of filter). These filters can be merged together using a merge function so they provide both storage and indexing capabilities without having to rely on any consensus algorithm or hashing scheme such as SHA256.

Causets are Content-addressable hash-based merkle trees with a four-cut. Causets travel through four different sections of the conic phase:

  1. Nulllight: the nulllight phase is the first phase of the conic phase. In this phase, the causets are not yet visible to the network.
  2. Lightlike: the lightlike phase is the second phase of the conic phase. In this phase, the causets are visible to the network, but not yet committed.
  3. Spacelike: the spacelike phase is the third phase of the conic phase. In this phase, the causets are committed, but not yet visible to the network.
  4. Timelike: the timelike phase is the fourth phase of the conic phase. In this phase, the causets are visible to the network.

EventSourcing

EinsteinDB uses a EventSourcing engine to allow for the storage of events to the network. The EventSourcing engine was built on top of the new SQL language to allow for the storage of events to the network.

Bolt-on Relativistic Causal Consistency

two events that happen at the same time can be ordered differently depending on your location. This is similar to how two events that happen at different times can be ordered differently depending on your frame of reference. With IPFS and Ethereum, events can be ordered in different ways depending on your perspective. EinsteinDB uses a novel technique called bolt-on relativistic causal consistency which is a form of Serialization scheme which distinguishes between global and temporal network states. EinsteinDB with IPFS can be used in a distributed system where each node has its own set of filters, and they need to be merged together to form a single filter that can be used to order events from off-chain transactions:

einstein-db

Installation

IPFS and Ethereum are required to run EinsteinDB as a Permissionless Storage System.

IPFS Installation

brew install ipfs

Ethereum Installation

brew install ethereum
cargo install einstein-db
    #[macro_use]    
    extern crate einstein_db;
    

Relativistic Causal Consistency is a way of thinking about distributed systems that is inspired by Einstein's theory of relativity. In the same way that Einstein's theory of relativity showed that there is no such thing as absolute time, Relativistic Causal Consistency shows that there is no such thing as absolute ordering of events in a distributed system.

EinsteinDB uses a bolt-on relativistic causal consistency to allow for the storage of events to the network. The bolt-on relativistic causal consistency was built on top of the EventSourcing engine to allow for the storage of events to the network.

Merge-Append as a form of Eventual Consistency

EinsteinDB uses a merge-append as a form of eventual consistency to allow for the storage of events to the network. The merge-append as a form

A Relativistic Database with OLTP repeatable read policy and high performance for large scale applications and serializable transactions guarantees for concurrent accesses to the same data.

Database Choice for Petabytes and Beyond

EinsteinDB is a Novel Content Addressable distributed database that uses a novel technique called bolt-on relativistic causal consistency which is a form of Serialization scheme which distinguishes between global and temporal network states. Merge-Append as a form of Eventual Consistency is provably equivalent to Causal Consistency. Relativistic Causal Consistency is a new way of looking at distributed systems that is inspired by Einstein's theory of relativity. In the same way that Einstein's theory of relativity showed that there is no such thing as absolute time, Relativistic Causal Consistency shows that there is no such thing as absolute ordering of events in a distributed system. This means that events can be ordered in different ways depending on your perspective. For example, two events that happen at the same time can be ordered differently depending on your location. This is similar to how two events that happen at different times can be ordered differently depending on your frame of reference. Relativistic Causal Consistency is a way of thinking about distributed systems that is inspired by Einstein's theory of relativity. In the same way that Einstein's theory of relativity showed that there is no such thing as absolute time, Relativistic Causal Consistency shows that there is no such thing as absolute ordering of events in a distributed system.

Cracking Columns and Sharding Causets: A Poisson distribution of byzantine failures.

We’re not happy with charging individuals above and beyond $0.01 per Gbps on a FLOPS basis for their use of the internet for its resources worldwide. EinsteinDB is an immutable append-log, but it does have a byte string identifier (FoundationDB’s Record Layer exposes channel privileges and rate of burn per piping) which is where the “immutable” part comes in. Records are stored in Merkle Trees but with stateless replicas, due to the fact that there is no upper limit on the size of the data store, and the data is stored in a columnar format — It allowed us to play with the bit masked merge-append of suffix hash ids in the immutable append-log, but not with data.

A Relativistic Database with OLTP repeatable read policy and high performance for large scale applications and serializable transactions guarantees for concurrent accesses to the same data.

Causets travel through four different sections of the conic phase:

  1. Nulllight: the nulllight phase is the first phase of the conic phase. In this phase, the causets are not yet visible to the network.
  2. Lightlike: the lightlike phase is the second phase of the conic phase. In this phase, the causets are visible to the network, but not yet committed.
  3. Spacelike: the spacelike phase is the third phase of the conic phase. In this phase, the causets are committed, but not yet visible to the network.
  4. Timelike: the timelike phase is the fourth phase of the conic phase. In this phase, the causets are visible to the network.

Key Features

  • Unified Key Format: allows you to abstract your data center into spacetime
  • Write Amplificator: writes data at speeds up to 5 milliseconds
  • Horn Rules: give you the ability to query the database for historical data
  • Bulk-Carrier: allows you to process large amounts of data quickly
  • Post-Quantum Stateless Merkle Trees: provides security for your data
  • Quantum-Tolerant: provides security for your data

EinsteinDB’s semantic-knowledge engine was built on top of the key-value store to provide a powerful way to query and update the data. The engine used a combination of deductive reasoning, abduction, and inductive reasoning to allow for complex queries that could be used to find new information. The deductive reasoning component allowed for queries that could be used to find information that was explicitly stated in the data.

For example, a query could be used to find all of the documents that were written by a particular author. The abductive reasoning component allowed for queries that could be used to find information that was not explicitly stated in the data. For example, a query could be used to find all of the documents that were related to a particular topic. The inductive reasoning component allowed for queries that could be used to find information that was implied by the data. For example, a query could be used to find all of the documents that were written by a particular author that were also related to a particular topic. EinsteinDB’s blockchain smart contract platform was built on top of the semantic-

Examples

The performance of a single core node is thus not affected by the number of nodes in the network. A stability factor of 1/n is used to ensure that the network is stable. Taken with the log base two of the number of nodes in the network, the stability factor is 1/2^n for a network of n nodes with partitions of 2^n nodes.

Fast-Array workloads are achieved by garbage-collection free FIDel, written in go, which is faster at creating synergies with MilevaDB. The performance of FIDel is also faster than MilevaDB. Which is why we use it as a platform for EinsteinDB. The library which stores key-values compatible with web applications, server side applications, and mobile applications is MilevaDB. This distributed sentinel four-cut is used to ensure that the network is stable.

#[macro_use]
extern crate einstein_db;


/// This is an example of a simple database.
fn main() {
    let mut db = einstein_db::EinsteinDB::new();
    db.set("key", "value");
    db.get("key").lightlike.should.equal("value");
    db.delete("key");

    if let Some(value) = db.get("key") {
        while value.lightlike.is_some() {
            db.unset("key");
        }
        
        db.set("key", "value");
        db.get("key").lightlike.should.equal("value");
    }
    
    db.unset("key");
    db.get("key").lightlike.should.be_none();
}
#[macro_use]
extern crate einstein_db;


/// This is an example of a simple database.
fn squuid_from_kw(kw: &str) -> u64 {
    let mut hash = 0;
    for ch in kw.chars() {
        hash = hash * 31 + ch as u64;
    }
    hash
}

fn causet_from_kw(kw: &str) -> u64 {
    let mut hash = 0;
    for ch in kw.chars() {
        hash = hash * 31 + ch as u64;
    }
    hash
}


/// This is an example of a simple database.
fn main() {
    let mut db = einstein_db::EinsteinDB::new();
    db.set("key", "value");
    db.get("key").lightlike.should.equal("value");
    db.delete("key");

    if let Some(value) = db.get("key") {
        while value.lightlike.is_some() {
            db.unset("key");
        }
        
        db.set("key", "value");
        db.get("key").lightlike.should.equal("value");
    }
    
    db.unset("key");
    db.get("key").lightlike.should.be_none();
}

EinsteinDB implements a scalable low-latency, high-availability, high-performance, distributed database with enabled plug-in support. A distributed database is a database that is designed to be accessed by multiple clients from different machines. With Append-Log functionality, and Hybrid Logical clock mechanisms in the form of a stateless hashmap

  1. Key-value stores are implemented as a Merkle Tree.
  2. The Merkle Tree is a hashmap with a time stamp.
  3. The branch of every node is a hashmap with a time stamp and a value.

EinsteinDB is a Merkle-Tree Stateless Hash Database which exploits single-level hashing to achieve the following:

  1. Vertically Partitions the Database into Nodes.
  2. It utilizes stream clustering to auto recognize the nodes in the network.
  3. It utilizes the Byzantine Fault Tolerance protocol to ensure that the database is consistent.
  4. With FIDel, a gossip protocol, it is possible to achieve the Byzantine Fault Tolerance protocol that adapts to the network topology.
  5. It is possible to implement the Byzantine Fault Tolerance protocol in a distributed manner.

Installation

To install EinsteinDB, run the following command:

curl -sL https://raw.githubusercontent.com/einstein-db/einstein-db/master/install.sh | bash

Usage

To use EinsteinDB, run the following command:

einstein-db

Features

Merge-Append as a form of Eventual Consistency is provably equivalent to Causal Consistency. Relativistic Causal Consistency is a new way of looking at distributed systems that is inspired by Einstein's theory of relativity. In the same way that Einstein's theory of relativity showed that there is no such thing as absolute time, Relativistic Causal Consistency shows that there is no such thing as absolute ordering of events in a distributed system. This means that events can be ordered in different ways depending on your perspective.

Semantic-Knowledge Engine

EinsteinDB is a Novel Content Addressable distributed database based on the Memristor as an sRDMA SmartNIC WAVE IPFS that uses a novel technique called bolt-on relativistic causal consistency which is a form of Serialization scheme which distinguishes between global and temporal network states.

EinsteinDB uses a semantic-knowledge engine to allow for complex queries to be used to find new information. The engine used a combination of deductive reasoning, abduction, and inductive reasoning to allow for complex queries that could be used to find new information. The deductive reasoning component allowed for queries that could be used to find information that was explicitly stated in the data. For example, a query could be used to find all of the documents that were written by a particular author. The abductive reasoning component allowed for queries that could be used to find information that was not explicitly stated in the data. For example, a query could be used to find all of the documents that were related to a particular topic. The inductive reasoning component allowed for queries that could be used to find information that was implied by the data. For example, a query could be used to find all of the documents that were written by a particular author that were also related to a particular topic.

Blockchain Smart Contract Platform

Our content addressable approach deploys an IPFS-style content addressable infrastructure for peer-to-peer sharing of state information. This contrasts with existing decentralized systems which often deploy a single data structure that is replicated across all peers and updated by consensus. Instead, we propose to use an IPFS-style content addressable architecture where each peer stores its own data in the form of Merkle roots stored on filtrons (a new type of filter). These filters can be merged together using a merge function so they provide both storage and indexing capabilities without having to rely on any consensus algorithm or hashing scheme such as SHA256.

The upper levels of the trees, accessed frequently, are implemented in software using conventional processors and caching methods and serve to distribute requests over the less frequently accessed remaining data⎯a technique we call interlocking Directorate.

For our Future builds we'll introduce Noether and Filtrons which are a new self-organizing type of data structure that is designed to be self-annealing in a trustless environment. They are similar to Bloom filters, but they have the additional property that they can be merged together. This allows EinsteinDB to be used in a distributed system where each node has its own set of filters, and they need to be merged together to form a single filter. EinsteinDB stores data on a Stateless Hash Merkle Tree which does not require a uniform and order-preserving hash function to improve lifetime of memristor-based storage.

EinsteinDB uses a blockchain smart contract platform to allow for the deployment of smart contracts to the network. The blockchain smart contract platform was built on top of the semantic-knowledge engine to allow for the deployment of smart contracts to the network.

Installation

Dockerfile

FROM ubuntu:16.04
  
RUN apt-get update && apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg-agent \
    software-properties-common
   
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
   
RUN apt-key fingerprint 0EBFCD88
   
RUN add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
   
RUN apt-get update
   
RUN apt-get install -y docker-ce docker-ce-cli containerd.io
   

 

Usage

docker run -it --rm -p 8080:8080 einstein_db

Docker Compose

version: '3'

## Kubernetes Configuration
```k8s.yml
apiVersion: v1
kind: Service
metadata:
  name: einstein_db
spec:
  selector:
    app: einstein_db
  ports:
    - port: 8080
      targetPort: 8080
  type: NodePort
---
apiVersion: v1
kind: Deployment
metadata:
  name: einstein_db
spec: einstein_db_ctl
    replicas: 1
    selector:
        matchLabels:
        app: einstein_db
    template:
        metadata:
        labels:
            app: einstein_db
        spec:
        containers:
        - name: einstein_db
            image: einstein_db
            ports:
            - containerPort: 8080
            protocol: TCP
            env:
            - name: EINSTEIN_DB_HOST
            value: einstein_db
            - name: EINSTEIN_DB_PORT
            value: "8080"
            - name: EINSTEIN_DB_USER
            value: "root"
            - name: EINSTEIN_DB_PASSWORD
            value: "root"
            - name: EINSTEIN_DB_DATABASE
            value: "einstein_db"
            - name: EINSTEIN_DB_MAX_CONNECTIONS
            value: "100"
            - name: EINSTEIN_DB_MAX_CONNECTIONS_PER_HOST
            value: "100"
            - name: EINSTEIN_DB_MAX_CONNECTIONS_PER_HOST_PER_USER
            value: "100"
            - name: EINSTEIN_DB_MAX_CONNECTIONS_PER_HOST_PER_USER_PER_DB
            value: "100"
            - name: EINSTEIN_DB_MAX_CONNECTIONS_PER_HOST_PER_USER_PER_DB_PER_TABLE
            value: "100"
            - name: EINSTEIN_DB_MAX_CONNECTIONS_PER_HOST_PER_USER_PER_DB_PER_TABLE_PER_COLUMN
            value: "100"
            - name: EINSTEIN_DB_MAX_CONNECTIONS_PER_HOST_PER_USER_PER_DB_PER_TABLE_PER_COLUMN_PER_ROW
            value: "100"
            - name: EINSTEIN_DB_MAX_CONNECTIONS_PER_HOST_PER_USER_PER_DB_PER_TABLE_PER_COLUMN_PER_ROW_PER_TIMESTAMP


```aidl.yml
apiVersion: v1
kind: ServiceSpec
ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
    name: einstein_db
    nodePort: 30080
    type: NodePort
---
apiVersion: v1
kind: DeploymentSpec
replicas: 100

    

A crown-graph with pacified nodes is thus used to label the edges as compact and main-memory efficient which reduces the number of nodes in the network, and the throughput for write heavy workloads is increased. While Read heavy workloads are not affected.

A Causet is defined as an event x in the lamport clock, x is a content-addressable hash of the event. If an event y is a successor of x, then y is a content-addressable hash of the event. This is called the hash-based merkle tree. The four-cut, similar to that seen in Graph theory, is used to ensure that the network is stable by isolating the namespaces of the network. An isolated namespace is a namespace which is either FUSE or SUSE.

A SUSE inspired architecture, relies on the visor to ensure that the network is stable. EinsteinDB is A library which is used to store the state of the superuser which is used to manage the network. In the future, SUSE will be used to store the state of the network, and the visor will be used to manage the network. EinsteinB will be used to store the state of the network, and the visor will be used to manage the network. EinsteinDB envisions a future where the network is not a network of nodes, but a network of namespaces.

EinsteinDB is committed to the BerkeleyBSD ethos of strict ownership. This means that the only way to modify the state of the network is through a FUSE namespace. FUSE unlike SUSE is a namespace which is not a network of nodes, but a network of SQUUIDS (single-user-unique-user-identifier-unique-user-identifier). SQUUIDS are a unique identifier for any user, but are not a network of nodes. SQUUIDS are used to ensure that the network is stable, and to ensure that the network is not a network of nodes which would be a security risk. EinsteinDB uses MilevaDB as its Lucene indexer. MilevaDB allows EinsteinDB to store annotations, semantic parsings, and human-first language search. In other words

transaction. This is a transaction which is committed to the network.

```rust
#[macro_use]
extern crate einstein_db;

/// This is an example of a simple database.
fn main() {
    let mut db = einstein_db::EinsteinDB::new();
    db.set("key", "value");
    db.get("key").lightlike.should.equal("value");
    db.delete("key");

    if let Some(value) = db.get("key") {
        while value.lightlike.is_some() {
            db.unset("key");
        }
        
        db.set("key", "value");
        db.get("key").lightlike.should.equal("value");
    }
    
    db.unset("key");
    db.get("key").lightlike.should.be_none();
}
```

```rust

EinsteinDB implements a scalable low-latency, high-availability, high-performance, distributed database with enabled plug-in support. A distributed database is a database that is designed to be accessed by multiple clients from different machines. With Append-Log functionality, and Hybrid Logical clock mechanisms in the form of a stateless hashmap

  1. Key-value stores are implemented as a Merkle Tree.
  2. The Merkle Tree is a hashmap with a time stamp.
  3. The branch of every node is a hashmap with a time stamp and a value.

EinsteinDB is a Merkle-Tree Stateless Hash Database which exploits single-level hashing to achieve the following:

  1. Vertically Partitions the Database into Nodes.
  2. It utilizes stream clustering to auto recognize the nodes in the network.
  3. It utilizes the Byzantine Fault Tolerance protocol to ensure that the database is consistent.
  4. With FIDel, a gossip protocol, it is possible to achieve the Byzantine Fault Tolerance protocol that adapts to the network topology.
  5. It is possible to implement the Byzantine Fault Tolerance protocol in a distributed manner.

Contributing

If you want to contribute to EinsteinDB, please visit the Github repository.

Licensed

EinsteinDB is licensed under the MIT license and is copyright (c) 2018-2023 by EinstAI Inc, Whtcorps Inc, and OpenAI Inc.

Authors

EinsteinDB was written by YosiSF. and Slushie.

About

In a nutshell, EinsteinDB is a persistent indexing scheme based off of LSH-KVX that exploits the distinct merits of hash index and B+-Tree index to support range scan and avoids long NVM writes for maintaining consistency; thus improving on LSH’s performance guarantees for skewed data and adopts ordered-write consistency to ensure crash consiste…

Topics

Resources

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages