Cube Creator is a tool to create RDF data Cubes (based on rdf-cube-schema) out of CSV files.
The tool is built with Typescript and is currently composed of two main parts: a Hydra API (built with hydra-box) and a Vue.js user interface. See the Architecture document for a high-level overview.
To be able to run transform/publish pipelines locally, an OIDC secret must be added to cli/.env as
AUTH_RUNNER_CLIENT_SECRET=foo-bar
It is obtained from keycloak
Alternatively, you can bypass authentication altogether by setting the environment variables in .env similar to the following example:
VUE_APP_E2E=true
VUE_APP_X_USER=john-doe
VUE_APP_X_PERMISSION=pipelines:read,pipelines:writeIf you have already started the application, make sure to run lando rebuild -y to apply the changes.
The easiest way it to start a local dockerized environment which will run the database, API and UI, and provide set up local HTTPS endpoints for them.
- Download and install lando
- it will install docker desktop if necessary
- Run
yarnto install packages - Run
lando startinside the repo- Docker daemon is also started automatically
- (Optional) Run
yarn seed-datato add sample projects to the database
Docker containers will start and the services will be available under the these URLs:
| Service | URL |
|---|---|
| API | https://cube-creator.lndo.site/ |
| UI | https://app.cube-creator.lndo.site/ |
| Fuseki | https://db.cube-creator.lndo.site/ |
| Minio | https://s3.cube-creator.lndo.site/ |
| Jaeger | http://jaeger.cube-creator.lndo.site |
| Prometheus | http://prometheus.cube-creator.lndo.site |
Lando uses its own Certificate Authority and it won't be trusted by your system. To trust the CA, follow the steps on https://docs.lando.dev/config/security.html#trusting-the-ca
This repository is using the GitHub-Flow. Hence all changes should be integrated using pull requests.
Commit messages usually follow the guidelines from Conventional Commits with the types in lower case.
To submit a bug or a feature request please create an issue in this repository.
All notable changes to the project(s) should be documented using @atlassian/changesets. When preparing a PR, run yarn changeset in the repository which will ask what changed, the affected packages, and create a small markdown file which must be committed to the repository.
Alternatively, that file can be created directly in the browser, courtesy of Changesets bot which creates PR comment summarising the changes included in the PR in question.
Releases are managed automatically by GitHub actions. As PRs get merged to master, the first job creates or updates a pull request which bumps package versions according to the combined information from all accumulated changeset files. Once merged, the second job kicks in which tags the repository and triggers INT deployment.
The package versions can also be bumped manually by running yarn changeset version in the repository and committing the result. When pushed, the new versions will be picked up by the aforementioned GitHub workflow job to tag the respository.
There are two types of e2e tests:
- API (Hydra): e2e tests that take the API as entrypoint
- UI (Cypress): browser-based e2e tests
Running the E2E tests can be done using: docker-compose run --rm e2e-tests, and docker-compose run --rm e2e-tests -- --grep pattern lets you select which tests to run.
For brevity, use npm script npm run test:e2e --grep pattern
We use Cypress to run UI e2e tests.
To simplify the tests, we circumvent authentication in the app. For that, the following variables need to be set in .env before running the UI:
VUE_APP_E2E=true
VUE_APP_X_USER=john-doe
VUE_APP_X_PERMISSION=pipelines:read,pipelines:write
We need a running instance of the app to test. The easiest way is to start lando: lando start
Then the following command can be used to run the tests interactively:
yarn --cwd ui test:e2e --url https://app.cube-creator.lndo.site
The --headless option allows running the tests without seeing the browser.