Synchronizes your data collection with GitHub and GitLab.
- Sync data collection from Git to DB
- Deal with multiple repositories
- Incremental updates
- Works with private or public repositories
Tuttle uses Gulp as its build tool which itself builds on NPM. To initialize the project and load dependencies run
npm i
Note: the
installcommands below assume that you have a local eXist-db running on port 8080. However the database connection can be modified in.existdb.json.
| Run | Description |
|---|---|
gulp build |
to just build Tuttle. |
gulp install |
To build and install Tuttle in one go |
The resulting xar(s) are found in the root of the project.
To run the local test suite you need an instance of eXist running on localhost:8080 and npm to be available in your path.
Run tests with npm test
Tuttle is configured in data/tuttle.xml.
@name is always the name of the destination collection. It will be configured in data/tuttle.xml
An example:
<repos>
<collection name="tuttle-sample-data">
<default>true</default>
<type>github</type>
<baseurl>https://api.github.com/</baseurl>
<repo>tuttle-sample-data</repo>
<owner>tuttle-sample-data</owner>
<token></token>
<ref>master</ref>
<hookuser>admin</hookuser>
<hookpasswd></hookpasswd>
</collection>
<collection name="tuttle-sample-gitlab">
<type>gitlab</type>
<baseurl>https://gitlab.com/api/v4/</baseurl>
<project-id>tuttle-sample-data</project-id>
<token>XXX</token>
<ref>master</ref>
<hookuser>admin</hookuser>
<hookpasswd></hookpasswd>
</collection>
</repos>Gitserver type: 'github' or 'gitlab'
- For github the baseurl is always api.github.com
- For gitlab the url can also be your private gitlab server egg 'https://gitlab.existsolutions.com/api/v4/'
- For github you have to specify the owner and the repo
- For gitlab you have to specify the project-id of the repository
Define the working branch of the git repository
If a token is specified Tuttle authenticates against GitHub or GitLab. When a token is not defined, Tuttle assumes a public repository without any authentication.
It is also possible to pass the token via an environment variable. The name of the variable have to be tuttle_token_ + collection (all dashes must be replaces by underscore). Example: tuttle_token_tuttle_sample_data
Be aware of the rate limits
- GitHub:
- 60 unauthenticated requests per hour
- 5,000 authenticated requests per hour
At this stage of development, the API keys must be generated via the API endpoint /git/apikey or for a specific collection /git/{collection}/apikey.
In the configuration tuttle.xml the "hookuser" is used to define the dbuser which executes the update.
Example configuration for GitHub:
- 'Payload URL': https://existdb:8443/exist/apps/tuttle/git/hook
- 'Content type': application/json
Example configuration for GitLab:
The dashboard can trigger a full deployment or an incremental update.
Full deployment clones the repository from git and install it as a .xar file.
With incremental update only the changes to the database collection are applied.
- customize the configuration (
modules/config.xql) - click on 'full' to trigger a full deployment from git to existdb
- now you can update your collection with a click on 'incremental'
Repositories from which a valid XAR (existing expath-pkg.xml and repo.xml) package can be generated are installed as a package, all others are created purely on the DB.
REMARK: Note that there may be index problems if a collection is not installed as a package.
REMARK: Only use it with data collections
The page below is reachable via api.html in your installed tuttle app.
Calling the API without {collection} config:default-collection() is chosen.
GET ~/tuttle/{collection}/git
With this most basic endpoint the complete data repository is pulled from the gitservice. The data will not directly update the target collection but be stored in a staging collection.
To update the target collection use another POST request to /tuttle/git.
The data collection is stored in /db/app/sample-collection-staging.
POST ~/tuttle/{collection}/git
The staging collection /db/app/sample-collection-staging is deployed to /db/app/sample-collection. All permissions are set and a pre-install function is called if needed.
POST ~/tuttle/{collection}/git
All commits since the last update are applied.To ensure the integrity of the collection, all commits are deployed individually.
GET ~/tuttle/{collection}/hash
Reports the GIT hashed of all participating collections and the hash of the remote repository.
GET ~/tuttle/{collection}/commits
Displays all commits with commit message of the repository.
GET ~/tuttle/{collection}/hook
The webhook is usually triggered by GitHub or GitLab. An incremental update is triggered. Authentication is done by APIKey. The APIKey must be set in the header of the request.
curl --header 'X-Gitlab-Token: RajWFNCILBuQ8SWRfAAAJr7pHxo7WIF8Fe70SGV2Ah' http://127.0.0.1:8080/exist/apps/tuttle/git/hook
GET ~/tuttle/{collection}/apikey
The APIKey is generated and displayed once. If forgotten, it must be generated again.
GET ~/tuttle/config
Displays the configuration and the state of the git repository.
States:
- uptodate: Collection is up to date with GIT
- behind: Collection is behind GIT and need an update
- new: Collection is not a tuttle collection, full deployment is needed
<tuttle>
<default>sample-collection-github</default>
<repos>
<repo type="github" url="https://github.com/Jinntec/tuttle-demo" ref="master" collection="sample-collection-github" status="uptodate"/>
<repo type="gitlab" url="https://gitlab.com/tuttle-test/tuttle-demo.git" ref="master" collection="sample-collection-gitlab" status="uptodate"/>
</repos>
</tuttle>POST ~/tuttle/{collection}/lockfile
Remove lockfile after anything goes wrong.
GET ~/tuttle/{collection}/lockfile
The running task is stored in the lockfile. It ensures that two tasks do not run at the same time.
To talk to the configured gitservice Tuttle needs an access token. These can be obtained from the respective service.
- see Creating a personal access token for github
- see Personal access tokens for Gitlab
The key for the gitservice must be configured in Gitservice configuration as shown above.
Will be implemented in release 2.0.0
Horace Parnell Tuttle - American astronomer
Archibald "Harry" Tuttle - Robert de Niro in Terry Gilliams' 'Brazil'