dynamodb log

sample repo to use DynamoDB with append only logs.

examples contain either Node.js or Python.

DynamoDB write triggers

EventBridge reads

Speakeasy JS

Here is the Speakeasy JS video from April 2021.

why?

I previously built level-eventstore and wanted the same benefits of append only logs, but using serverless.

By using DynamoDB with DynamoDB Streams we can build append only logs. Although we cannot implement a strict append only log we can order log items by when they are written, and with conditional writes we can perform optimistic locking.

We can preserve the item order when reading from DynamoDB using a sort key.

• Logs are saved in DynamoDB.
• Publish / Subscribe changes using EventBridge.

license

Apache License, Version 2.0

writing to DynamoDB

example payload written to DynamoDB

{
  "pk": "users#12#stream",
  "sk": 1,
  "type": "signup",
  "log": "users",
  "payload": {
    "id": "12",
    "name": "test",
    "email": "test@example.com"
  }
}

• pk (partition key) is log name#id#stream.
• sk (sort key).
• type is the name of the event useful for event handlers.
• log name of log.
• payload should contain the id and any optional extra fields.

When items are written to DynamoDB they are written to the DynamoDB stream for the shard they belong to in the order they are written.

The lambda is then triggered which will publish the changed keys to EventBridge.

example lambda triggers

• Node.js lambda trigger
• Python lambda trigger

example event handlers triggered by EventBridge

• Node.js event handler example
• Python event handler example

install prerequisites

only required if using python

• python 3.8.6
• virtualenv

required to run locally in offline mode and for linting

• docker
• docker-compose

required (serverless framework and tools)

• nodejs 14

cli.sh

interactive cli script for both local and AWS

npm install

./cli.sh

Speakeasy JS

Speakeasy JS

  npm run speakeasyjs

in AWS

export AWS credentials before running cli.sh

Node.js deploy to AWS

npm i
./cli.sh
npm exec sls deploy -- --stage dev -c serverless-node.yml

Node.js remove stack in AWS

npm i
./cli.sh
npm exec sls remove -- --stage dev -c serverless-node.yml

Python deploy to AWS

npm i
./cli.sh
rm -rf venv
virtualenv venv
. venv/bin/activate
pip3 install -r requirements.txt
npm exec sls deploy -- --stage dev -c serverless-python.yml
rm -rf venv

Python remove stack in AWS

npm i
./cli.sh
npm exec sls remove -- --stage dev -c serverless-python.yml

Query DynamoDB

./cli.sh
./dynamodb-local-query.sh

generate fake user data

Generate fake user data locally or in aws.

./create-fake-data.sh

offline

Node.js

./cli.sh
npm exec sls offline start -- --stage=local -c serverless-node.yml

Python

./cli.sh
virtualenv venv
. venv/bin/activate
pip3 install -r requirements.txt
npm exec sls offline start -- --stage=local -c serverless-python.yml

Add item using aws cli.

./cli.sh
export AWS_ACCESS_KEY_ID=x
export AWS_SECRET_ACCESS_KEY=x
export AWS_DEFAULT_REGION=us-east-1
aws dynamodb put-item \
  --table-name local-dynamodb-logs \
  --item """
  {
    \"pk\": { \"S\": \"users#12#stream\" },
    \"sk\": { \"N\": \"${sk:-1}\" },
    \"type\": { \"S\": \"signup\" },
    \"log\": { \"S\": \"users\" },
    \"payload\": { \"M\": {
      \"id\": { \"S\": \"12\"},
      \"name\": { \"S\": \"test\"}
      \"email\": { \"S\": \"test@example.com\"}
    }}
  }""" \
  --condition-expression "attribute_not_exists(pk)" \
  --endpoint http://localhost:8000

Query DynamoDB

./cli.sh
./dynamodb-local-query.sh

Cleanup

Remove dynamodb data and docker-compose processes.

Also needed if you have new AWS credentials.

./cli.sh stop

retries offline and in AWS

offline retries

per stack retry limit is configurable using the offline eventbridge plugin.

custom:
  serverless-offline-aws-eventbridge:
    retryDelayMs: 1000
    maximumRetryAttempts: 5

destinations

using onFailure failures can be handled by your application.

failure payloads look like this

eventHandler:
  handler: src/handler.handler
  destinations:
    onFailure: retry
  iamRoleStatements:
    - Effect: Allow
      Action:
        - "lambda:InvokeFunction"
      Resource:
        - !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:${AWS::AccountId}:function:dynamodb-logs-${opt:stage}-retry"

{
  "version": "1.0",
  "timestamp": "<timestamp>",
  "requestContext": {
    "requestId": "<uuid>",
    "functionArn": "arn:...",
    "condition": "RetriesExhausted",
    "approximateInvokeCount": 3
  },
  "requestPayload": {
    "version": "0",
    "id": "<uuid>",
    "detail-type": "stream changes",
    "source": "dynamodb-log",
    "account": "<account>",
    "time": "<timestamp>",
    "region": "<region>",
    "resources": [],
    "detail": {
      "log": "<log>",
      "key": {
        "pk": "<pk>",
        "sk": 0
      },
      "type": "<signup>",
      "payload": {
        "id": "<id>"
      }
    }
  },
  "responseContext": {
    "statusCode": 200,
    "executedVersion": "$LATEST",
    "functionError": "Unhandled"
  },
  "responsePayload": {
    "errorType": "Error",
    "errorMessage": "<error message>",
    "trace": []
  }
}

replay using EventBridge archive

When run offline events are persisted to sqlite3 file ./db.

Events can be replayed to EventBridge or DynamoDB, you can also replay events from AWS to local which will deplay an extra stack using a websocket lambda and a new event target that writes to the connected websockets.

./cli.sh
./run-archive.js