DCAT DRY

DCAT-AP Dataset Relationship Indexer. Indexing linked data and relationships between datasets.

Features:

• index a distribution or a SPARQL endpoint
• extract and index distributions from a DCAT catalog
• extract a DCAT catalog from SPARQL endpoint and index distributions from it
• generate a dataset profile
• show related datasets based mainly on DataCube and SKOS vocabularies
• indexing sameAs identities and related concepts

Build & run with Docker

For DCAT-DRY service only:

docker build . -t dcat-dry
docker run -p 80:8000 --name dcat-dry dcat-dry

For the full environment use docker-compose:

docker-compose up --build

Build & run manually

CPython 3.8+ is supported.

Install redis server first. In following example we will assume it runs on localhost, port 6379 and DB 0 is used.

Setup postgresql server as well. In the following example we will assume it runs on localhost, port 5432, DB is postgres and user/password is postgres:example

You will need some libraries installed: libxml2-dev libxslt-dev libleveldb-dev libsqlite3-dev and sqlite3

Run the following commands to bootstrap your environment :

git clone https://github.com/eghuro/dcat-dry
cd dcat-dry
poetry install --with robots,gevent --without dev
# Start redis and postgres servers

# Export environment variables
export REDIS_CELERY=redis://localhost:6379/1
export REDIS=redis://localhost:6379/0
export DB=postgresql+psycopg2://postgres:example@localhost:5432/postgres

# Setup the database
alembic upgrade head

# Run concurrently
celery -A tsa.celery worker -l debug -Q high_priority,default,query,low_priority -c 4
gunicorn -w 4 -b 0.0.0.0:8000 --log-level debug app:app
nice -n 10 celery -l info -A tsa.celery beat

In general, before running shell commands, set the FLASK_APP and FLASK_DEBUG environment variables :

export FLASK_APP=autoapp.py
export FLASK_DEBUG=1

Deployment

To deploy:

export FLASK_DEBUG=0
# Follow commands above to bootstrap the environment

In your production environment, make sure the FLASK_DEBUG environment variable is unset or is set to 0, so that ProdConfig is used.

Shell

To open the interactive shell, run :

flask shell

By default, you will have access to the flask app.

Running Tests

To run all tests, run :

flask test

Before execution

# Prepare couchdb :

curl -X PUT http://admin:password@127.0.0.1:5984/_users
curl -X PUT http://admin:password@127.0.0.1:5984/_replicator
curl -X PUT http://admin:password@127.0.0.1:5984/_global_changes

# Migrate database :

alembic upgrade head

API

To start batch scan, run :

flask batch -g /tmp/graphs.txt -s http://10.114.0.2:8890/sparql

Get a full result :

/api/v1/query/analysis

Query a dataset :

/api/v1/query/dataset?iri=http://abc