Private Data Service Reference Implementation

This provides an easy mode installation for the private data service. It will install docker, docker-compose, provide a sane default configuration, and start all related services.

Grab the files

As root, run the following in a terminal:

curl -L https://github.com/skuid/pdsref/archive/master.tar.gz | tar --strip-components=1 -xzvf - -C /opt/skuid

Usage

Make sure you answer y to both make install and make start. The scripts aren't smart enough yet to know if you've chosen to install the local postgres image or not.

Install dependencies and set environment

make install will install the dependencies (like docker and docker-compose). It will also prompt for the database credentials for the Warden and Clortho persistence stores. Part of the install is to set the database encryption key. It should not overwrite this value.

make install

Start it up

Running make start will pull and start all of the docker images that make up the data services. These images will use the environment files created during make install.

make start

Test it out

You should see a 200/OK response from:

curl http://localhost/ping

And from this route:

curl http://localhost:3005/warden/ready

{"status": "ready"}

Updating

When there are new versions they will be changed in docker-compose.yml. To get the latest versions applied, just pull the latest from this repo and run make upgrade

curl -L https://github.com/skuid/pdsref/archive/master.tar.gz | tar --strip-components=1 -xzvf - -C /opt/skuid
cd /opt/skuid
make upgrade

Using a local postgres docker image for Warden and Clortho

If you would rather run two PostgreSQL docker images for the Warden and Clortho persistence stores, you can move env/warden and env/clortho (if they already exist) and run the following commands. make dbinstall will create the env files for Clortho and Warden. make dbstart will pull the images, start them up, and then wait for the databases to be ready.

make dbinstall
make dbstart

Architecture

Here are some architecture diagrams.

Proof of concept

This is a quick POC way of running the Private Data Service (PDS) locally on an EC2 instance (or other VM). Running PDS in this way in a production setting is not advisable. We follow the 12 Factor App principles and will fail fast for unexpected error condtions with the expectation that the service will be restarted by an orchestration layer like Kubernetes.

Reference

This is a conceptual diagram showing how PDS might be run in production and is similar to how we run the data service microservices in our cloud offering, using Kubernetes as the orchestration layer, log aggregators, and monitoring (which are not pictured here).

Clean everything

Danger: this will delete all of your data. Use with caution

If you would like to completely start over, you can run:

make clean

This will stop and remove all of your running docker images, including ones that were not created by this script. Hopefully we'll add some more smarts to limit it to just our docker images in the future, but for now, beware. It will also drop your database data and remove the settings in your environment file.

Advanced mode

If you want to use your own postgres database, answer N to the two questions above. During the install it will prompt for the postgresql database information. During make start, it will not start a local postgres instance and will instead rely on the values you provided.

Useful commands for diagnosing problems (and other things)

To enter psql against clortho or warden (substituting out the appropriate environment variable file):

# Connect to clortho. Use a different environment variable file with the
# PG[HOST|USER|DATABASE|PASSWORD] environment variables for the instance
# to which you want to connect
docker run --rm --network skuid_pds --env-file /opt/skuid/env/clortho -it postgres:9.6 psql

To run pg_dump:

# Dump warden's database
docker run --rm --network skuid_pds --env-file /opt/skuid/env/warden postgres:9.6 pg_dump warden > warden_bak.sql

To restore a database using the docker image:

# Restore warden's database
docker run --rm --network skuid_pds --env-file /opt/skuid/env/warden -v "/opt/skuid:/opt/skuid" -it postgres:9.6 psql -f /opt/skuid/warden_bak.sql

Connect to a running container:

# Connection to warden. Find the container name using 'docker ps'
docker exec -ti skuid_warden_1 /bin/sh