Skip to content

willemt/pearldb

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

image

image

image

What?

PearlDB is a durable HTTP Key-Value pair database. It uses LMDB for storing data, and H2O for HTTP.

PearlDB is completely written in C.

Persistent connections and pipelining are built-in.

PearlDB uses bmon to batch LMDB writes.

Goals

  • Speed
  • Low latency
  • Durability - An HTTP response means the write is on disk
  • Simplicity outside (RESTful inteface)
  • Simplicity inside (succinct codebase)
  • HTTP caching - Because the CRUD is RESTful you could hypothetically use an HTTP reverse proxy cache to scale out reads. You could use multiple caches to create an eventually consistent database

Ubuntu Quick Start

Usage

Examples below make use of the excellent httpie

Starting

Get

We obtain a value by GET'ng the key.

In this case the key is "x". But we get a 404 if it doesn't exist.

You MUST specify a path.

Put

We use PUT for creating or updating a key value pair. PUTs are durable - we only respond when the change has been made to disk.

PUTs have an immediate change on the resource. There is full isolation, and therefore no dirty reads.

Now we can finally retrieve our data via a GET:

The slash at the end is optional.

The user must specify the capacity of the database upfront. PearlDB does not support automatic resizing. A PUT will fail if it would put the database over capacity.

You can't PUT under nested resources.

Put without a key (POST)

If you want PearlDB to generate a key for you, just use POST.

The Location header in the response has the URI of the newly created resource. The URI is the URL safe base64 encoded UUID4.

Providing a URL (ie. key) with POST doesn't make sense, and will result in a 400.

Get keys

You can get the keys that match a prefix by using the /key/XXX/ nested resource.

Without a prefix you get all keys.

Existence Check

To check for existence use the HEAD method. This is great, because PearlDB doesn't waste bandwidth sending the document body.

Delete

DELETEs are durable - we only respond when the change has been made to disk.

Of course, after a DELETE the key doesn't exist anymore:

Compare and Swap (CAS)

A form of opportunistic concurrency control is available through ETags.

When the client provides the Prefers: ETag header on a GET request we generate an ETag. A client can then use the If-Match header with the ETag to perform a conditional update, (ie. a CAS operation). If the ETag has changed then the PUT operation will fail. CAS operations are great because there is no locking; if a CAS operation fails for one client that means it has succeeded for another, ie. there has been progress.

Imagine two clients trying to update the same key. Client 1 requests an ETag. The ETag is provided via the etag header.

If client 1 requests an ETag again, the same ETag is sent:

Client 2 does a PUT on x. This will invalidate the ETag.

Client 1 uses a conditional PUT to update "x" using the If-Match tag. Because the ETag was invalidated, we don't commit, and respond with 412 Precondition Failed.

Once this happens we can retry the PUT after we do a new GET.

The PUT will succeed because the ETag is still valid.

However, if we use the ETag again it will fail.

Notes about ETags:

  • On reboots, PearlDB loses all ETag information
  • On launch PearlDB generates a random ETag prefix
  • ETags are expected to have a short life (ie. < 1 day)

OPTIONS

You can check what HTTP methods are available to a resource using the OPTIONS method. This is useful as some systems like HAProxy use the OPTIONS method as a healthcheck.

Shutting down

Building