Releases: INCATools/ontology-development-kit
ODK 1.5 (March 2024)
We thank the community for all your support and feedback! Without your issues and contributions we would not be able to do our work!
Update instructions
https://oboacademy.github.io/obook/howto/odk-update/
Highlights and possibly breaking changes (Read this!)
- All processes within the ODK container now runs under the identity of an (unprivileged) user by default. This fixes the issue of generated files being owned by the superuser, when the Docker daemon itself runs as the superuser (as is the case by default on most GNU/Linux systems). See PR1, PR2, PR3.
- Consequences: Some workflows that require superuser rights may not work anymore as expected.
- Mitigation: If you have a workflow that requires being run as a superuser (for example, if you need to install extra Debian/Ubuntu packages via
apt-get
), set the environment variableODK_USER_ID
to 0 when running that workflow or, to always run all workflows as a superuser (as was the case in previous ODK versions), set the configuration parameterrun_as_root
toTrue
before updating your repository.
- Change the definition of "base release". See PR.
- Consequence: The base file now does not only contain the editors axioms in their raw form, but the axioms "as intended by the ontology developer", for example, including inferences. For the base-specification see here.
- Mitigation: If you want a release that corresponds exactly to the old base file, use
baselite
instead.
- Allow refreshing the mirrors (externally downloaded ontologies) under
IMP=false
. See PR.- Consequence: you now cannot rely on
IMP=false
anymore if you want to avoid refreshing mirrors as well - you need to useIMP=false MIR=false
instead!
- Consequence: you now cannot rely on
- Make qc.yml ODK managed by default, so it is actually being updated along with the rest of the files. See PR.
- Consequence: If you have overwritten that workflow with custom content, you have to follow the mitigation strategies.
- Mitigation: (1) migrate the custom content to a differently named workflow or (2) deactivate syncning of that workflow manually by adding a
workflows
section to your ODK config.
- Generate
custom_reports
during release process PR.- Consequence: This means that all reports configured in ODK are automatically updated at release time, so you have more files need to review during a release.
- Mitigation: Add your reports to
.gitignore
or remove them from your ODK config.
- We added the "International Edition" as a release product, including an entire workflow system for managing translations using the Babelon format. This feature is still under development, but works - feel free to reach out if you like to test it.
New and updated tooling
- New ROBOT Version 1.9.5 was added in a previous minor release.
- A lot of updated python tools, including OAK (0.5.25), SSSOM tools (0.4.4), LinkML (1.7.5), kgx (2.2.5) and curies (0.7.7). A full list of all available python tools and their versions is available here.
- The Babelon Toolkit (0.2.7) for managing multilingual ontologies has been added as well.
- Added table-reader plugin for mkdocs to directly integrate CSV files into your ontology documentation, see PR.
- Added Perl module Business::ISBN, see PR.
- Updated Apache Jena (4.9.0), Soufflé, Fastobo-validator, Ammonite REPL & Script-Runner (2.5.9), DOSDP-Tools (0.19.3), Relation-Graph (2.3.2).
- Added SSSOM-Java (0.7.5) (PR) and KGCL-Java (0.3.2) (PR) to
odkfull
.
New configuration options
- Add option to include defined-by annotation in imports, see PR.
- Add option
release_annotate_inferred_axioms
, see PR, to enable the annotation of inferred axioms during release.
Makefile workflows
- Make it possible to collect per-command resource usage data, see PR. This allows us to see which goals consume how much memory, and how long they take, to identify performance bottlenecks.
- Enable support for custom ROBOT plugins in ODK, see PR
- Add a
test_fast
goal to allow running tests without refreshing dependencies PR. - Re-integrate LightRDF RDF/XML validation, see PR1, PR2
- Add SPARQL Check to find uses of deprecated DC PR
- Add release diff action, see PR. This allows posting diffs automatically as a comment to a Pull Request.
- Add convenience check if customised ROBOT report config is out of date PR. This allows checking if you are missing out on some new ROBOT report checks!
- Add a nicer, more comprehensive way to understand the versions of the tools used in ODK, see PR.
Runner and Infrastructure
- Allow passing configuration options to the OWL API, see PR.
- Automatically check whether the repository needs to be updated, see PR.
- Forward the host SSH agent socket into the container, see PR.
Bugfixes
- Fix import to use
OBOBASE
in base-iri, notURIBASE
, see PR. - Do not create individual import modules when
use_base_merging
is enabled, see PR. - Make docs workflow configurable, see PR.
- Update
illegal-date-violation.sparql
to acceptxsd:dateTime
, see PR. - Update URL to show CI status badge correctly on repo README.md, see PR.
For more detailed overview of the changes see:
ODK 1.4.3 (September 2023 2nd Edition)
This is a point release of the Ontology Development Kit. It uses updated versions of several components, including:
- ROBOT v1.9.5
- Ontology Access Kit (
oaklib
) 0.5.20 - Python
curies
packages 0.6.4 - Python SSSOM toolkit 0.3.41
This release does not change anything to the ODK-managed workflows, so running sh run.sh make update_repo
to adapt your repositories to this new release is not needed. Just make sure to use the latest
image by running docker pull obolibrary/odkfull
.
ODK 1.4.2 (September 2023)
This is a point release of the Ontology Development Kit that brings up-to-date versions of several components:
- Apache Jena updated to version 4.9.0
- Soufflé updated to version 2.4
- Fastobo-validator updated to version 0.4.6
- Ammonite updated to version 2.5.9
- Several Python packages updated, including:
- oaklib 0.5.17
- sssom 0.3.40
- sssom-schema 0.15.0
- linkml 1.5.7
- /!\ The
bioregistry
dependency is removed in favour of thecuries
package
To use this release, run docker pull obolibrary/odkfull on your machine.
Updating your individual ODK-managed repositories with make update_repo is not needed as this release does not bring any change to the standard workflows.
ODK 1.4.1 (May 2023)
This is a point release of the Ontology Development Kit that brings up-to-date versions of several components:
- ROBOT version 1.9.4;
- Ontology Access Kit version 1.5.6;
- Bioregistry version 0.9.15;
- LinkML version 1.5.2.
To use this release, run docker pull obolibrary/odkfull
on your machine.
Updating your individual ODK-managed repositories with make update_repo
is not needed as this release does not bring any change to the standard workflows.
Full Changelog: v1.4...v1.4.1
ODK 1.4 (February 2023)
Update instructions
https://oboacademy.github.io/obook/howto/odk-update/
Changes from ODK 1.3.1 to 1.4
A full record of all changes can be seen here.
New and updated tooling
- New ROBOT Version 1.9.3
- A lot of updated python tools, including OAK (0.1.71), SSSOM tools (0.3.22), LinkML (1.4.4) and curies (0.4.2).
gh
is now installed in ODK, which means that GitHub workflows can be run directly through ODK. For example, try out the newpublic_release
workflow which automatically creates a GitHub release for you.- A full list of all available python tools and there vesions is available here
New configuration options
- Making the
uribase
configurable. You can now set the URI base in yourmyont-odk.yaml
file to something different from http://purl.obolibrary.org/obo/, which enables developers from outside OBO to use ODK. Note that there is no guarantee that you can export your ontology to the OBO file format of you customise yourbaseuri
this way! import_component_format
: You can now configure the format in which your imports and components are serialised. For example, rather than owl (RDFXML), which used to be the default, you can set this option to ofn. For a complete reference see https://robot.obolibrary.org/convert.mirror_type
: You can define the type of the mirror for each import. Supported: base, custom, no_mirror.release_use_reasoner
: If false, no reasoning is performed for generating release files. This is only relevant for building application ontologies, where all components are already fully classified.sparql_test_on
: You can define the list of input files to run the custom SPARQL queries. Supported: edit, and any release artefact, e.g myont-base.owl (example).use_edit_file_imports
: If TRUE, use whatever imports you have in the edit file to create the release (default). If FALSE, components (and imports) are merged into the release independent of whether they are mentioned in the edit file or not. This can help for example in cases where import modules are so large, they cannot be checked into version control.- The
ci
option now takesgitlab-ci
as a value, which sets your repo up with basic Continuous Integration Testing for GitLab.
Makefile workflows
- Adding new
make reason_test
command (pull, issue) - Adding intermediate artefact
$(TMPDIR)/$(ONT)-preprocess.owl
to the release workflow which enable the centralisation of preprocessing in a single make step. Basically, rather than creating release artefacts directly from the editors file (e.g. bfo-edit.owl), we add intermediate step from which all releases are derived. This intermediate can then be customised by the user (pull, issue) - Introduces an experimental new release called "base-plus", which includes the inferred and non-redundant classification of the ontology. This is highly experimental and may be removed in a future release of ODK. The new goal is much more rigorous in removing axioms from other ontologies as well. The old base file can now be exported as an "editors release" instead (pull, issue).
- Introduces the option of skipping the use of reasoner during the release process (important for application ontologies), (pull, issue)
- Introduces a new mode that enable the use of ODK entirely without owl:imports in the edit file (this is great in case we want to use the ODK workflows but not check in any files, imports or components, into version control, like huge application ontologies), (pull, issue).
- Adds a feature to directly support ROBOT templates in components (example). Templates need to be activates with the project-level
use_templates: TRUE
option, and can then be used to define components, like in the example. - Adds an option to do a
public_release
on Github which creates a GitHub release, tags it, and uploads release artefacts. - Adds a
release_diff
workflow which creates a simple markdown report between the latest release and the current one. - Adds a feature to directly support SSSOM mapping files, both in components and as standalone (example). This allows to curate, for example, SSSOM mapping files in tables and them release them as part of the general release process. It also allows extracting mappings from inside the Ontology automatically using the SSSOM toolkit.
- Changes in the
src/ontology/run.sh
wrapper script:- It is now possible to execute the Docker image through Singularity.
- The
IMAGE
variable, which can used to specify an alternative ODK image, has been renamed toODK_IMAGE
. - A new variable
ODK_TAG
has been introduced, allowing to specify an alternative tag (default islatest
). A tag may also be specified directly as part of theODK_IMAGE
variable (as inODK_IMAGE=odkfull:v1.3.1
). - A new variable
ODK_BINDS
has been introduced, allowing to specify extra bindings between a directory on the host computer and a directory inside the Docker container. - Variables used by the
run.sh
script can now be set in asrc/ontology/run.sh.conf
file, which will be sourced by the wrapper script. - With the config file option
use_env_file_docker
, if true, you can pass your local environment to the docker container. It is strongly advised to addrun.sh.env
to.gitignore
before using this feature. Committing your environment to git may reveal passcodes and other private information!
The GitHub issues dealt with as part of this release
- Fix redundant filter in sparql queries by @anitacaron in #633
- Add docker requirement to top-level README by @kltm in #635
- Update to latest relation-graph by @balhoff in #634
- Make run.sh Java Option handling a bit nicer by @matentzn in #616
- Enable use of singularity as an option by @pcm32 in #640
- Allow to only push
dev
-tagged images to Docker Hub. by @gouttegd in #653 - Schema config documentation by @hkir-dev in #656
- Rename dosdp-tools directory. by @gouttegd in #652
- Rename IMAGE to ODK_IMAGE in the run.sh script. by @gouttegd in #655
- Build
odkfull:dev
fromodklite:dev
. by @gouttegd in #661 - Update constraints from within the
odkbuild
image by @gouttegd in #663 - Add constraints.txt make goal by @matentzn in #662
- Adding a preprocessing intermediate file to the ODK build process by @matentzn in #639
- Make sure the Python package
six
is present inodklite
. by @gouttegd in #666 - Make Base IRI configurable by @matentzn in #617
- Update constraints.txt by @github-actions in #664
- Update catalog-v001.xml.jinja2 to reflect base IRI changes (bugfix) by @matentzn in #668
- Fix wrong config for recreating components by @matentzn in #670
- fixed parentheses in inject queries by @ehartley in #672
- Update constraints.txt by @github-actions in #667
- Update constraints.txt by @github-actions in #673
- Allow extra volume bindings and custom wrapper config file by @gouttegd in #678
- Update constraints.txt by @github-actions in #681
- Add documentation about custom sparql checks by @anitacaron in https://github.com/INCATools/ont...
June 2022 Release
Official Changelog:
- Update to ROBOT 1.9.0, see ROBOT release notes. One major change concerning ODK: the OBOGraphs JSON serialiser has been updated significantly, which means obographs json files may look a bit different. Most important change: empty elements (xrefs) are no longer serialised.
- Added shortcut make commands for recreating components:
recreate-components
andrecreate-%
(pull). - Added new shortcut command for fast releases:
make prepare_release_fast
. - New tools:
- Ontology Access Kit, a python library for interacting with ontologies, the the ODK (pull).
- gh: GitHub CLI for running releases, createing pull requests and more
- j2cli: Jinja template client
- rdftab: RDF Tables with Rust
- gizmos: Python utilities for ontology development
- Other: obographviz, node.js, graphviz, npm
- The ontology diff GitHub action, a tool that allows you to create diffs for you ontology pull requests, is now automatically synced with your ODK setup (pull).
- Added
workflows
option to ODK config. This allows users to choose which workflows (at the moment GitHub actions-based) should be automatically synchronised. - Custom ROBOT SPARQL reports now consider the imports by default (pull)
- Base IRIs can now be configured for ROBOT report, similar to what OBO Dashboard does
- Technical
- Run internal
make
and $(MAKE) to ensure that parameters are propagated correctly (issue). - Mirror goal is only run when mirrors have been downloaded (pull)
- src/ontology/run.sh can now be configured with a ODK_JAVA_OPTS parameter at runtime, rather than just with a
robot_java_args
option in the projects ODK config file. This allows to more easily run releases across environments with different levels of memory. Example:ODK_JAVA_OPTS=-Xmx4G ODK_DEBUG=yes ./run.sh make odkversion
- src/ontology/run.sh warns the user in certain cases to update their local ODK image.
- Better linebreaks in Makefile to make lines more.
- Run internal
- Documentation
- Added improved
CONTRIBUTING.md
to ODK repo template - Removed history.md page from ODK documentation
- Some improved settings to default mkdocs documentation pages, such as better search, edit buttons and more.
- Some improved documentation on components (pull)
- All ODK-related general docs now moved to https://oboacademy.github.io/obook.
- DOSDP documentation pages are now only created when dosdp is configured.
- Added improved
- Bugs
- The automatic docs workflow now correctly
uses: mhausenblas/mkdocs-deploy-gh-pages@master
. - In ODK 1.3.0
prepare_release
depended onall
, which caused some cyclicity for some customised setups. This is now changed to a separateall_odk
goal. - Simple ontology release artefacts now correctly depend on the
simple seed
. - Similar to subset declarations, ODK now injects synonymtype declarations when extracting a module.
- diff.yaml no longer added by default, has to be configured.
- The automatic docs workflow now correctly
For changes on previous versions see complete CHANGELOG
Automated GitHub Changelog:
- Make sure diff.yml action is automatically copied during update by @shawntanzk in #564
- Update dosdp-tools to 0.19.2 by @balhoff in #575
- added components doc by @shawntanzk in #577
- Removed files that are repeated in obook by @shawntanzk in #578
- Add oaklib by @gouttegd in #586
- First fixes 131: components, $(MAKE) and reports by @matentzn in #570
- Build rdftab from source. by @gouttegd in #593
- Improving run.sh.jinja2 by @matentzn in #594
- Adding missing ontology annotations on imports by @matentzn in #597
- Update dosdp-tools version by @balhoff in #596
- issue 589 by @cmungall in #590
- Add ontodev-gizmos, psycopg2, pyspellchecker. by @gouttegd in #602
- Fix argument passing in the Windows run.bat script. by @gouttegd in #603
- ont.is_large only makes sense with module type slme by @matentzn in #607
- Upgrade base system by @gouttegd in #604
- Add ability to set base IRIs for report in ODK by @matentzn in #609
- [WIP] Update contribution guidelines by @cthoyt in #598
- Add Robot 1.9.0 to Dockerfile by @matentzn in #621
- Removing History page and add nice default config for mkdocs by @matentzn in #611
- New contribution guide by @matentzn in #620
- Add break lines to makefile by @anitacaron in #612
New Contributors
Full Changelog: v1.3.0...v1.3.1
Feburary 2022 Release (1.3.0)
v1.3.0 (24 February 2022)
As always, update your local ODK with docker pull obolibrary/odkfull
.
Breaking change for users of DOSDP in CI (GitHub Actions)
A new tool was introduced for users of DOSDP for better validation. See this comment about updating your CI pipeline.
New features:
- New DOSDP workflow for matching patterns: This is a powerful workflow that "matches" your DOSDP YAML templates against your ontology to actually create TSVs - so the other way around of what we usually do.
- To set this up, you introduce a new "pattern_pipelines_group" called
matches
, which can then be invoked withsh run.sh make dosdp-matches-%
to match all configured DOSDP tempates with the ontology. - Example see here.
- To set this up, you introduce a new "pattern_pipelines_group" called
- Experimental feature-diff feature: When creating a pull request, the ODK can now automatically make a comment to the pull request informing editors of changes to the ontology (issue)
- Adding Relation Graph to ODK (pull request, pull request)
- New QC checks:
- IRIs in labels: Often when using autogenerated labels in DOSDP we do not realise we are accidentally missing labels to for generating class names. We introduced a QC check to protect against that (pull request, issue)
- Ensuring that the range of never_in_taxon:, present_in_taxon: , oboInOwl:inSubset, rdfs:seeAlso, foaf:depicted_by and dcterms:contributor is always IRI (issue, pull request)
- Switched the default Ontology documentation theme from
readthedocs
tomaterial
(pull request) - New DOSDP tools version 0.19.1
Bugfixes:
- Many faulty dependencies in the Makefile that triggered unnecessary re-runs of make were removed (issue)
- Some major refactoring of the main
Makefile
(pull request). This includes, in particular, a smarter way to deal with web-dependencies (mirrors, dosdp-templates, components). - Fixes to documentation (Managing dependencies with base files, Import seed system)
- Some fixes to DOSDP pipeline setup (issue)
- Git
main
branch was hardcoded tomaster
during the seeding process, now uses the ODK config file (pull request, issue) - Making sure the repo seeding process preserves file privileges. This ensures, in particular, that
run.sh
is executable and can be run using./run.sh
instead ofsh run.sh
(issue, pull) - DOSDP Validation now uses python up to date library
- Auto-deployment of ODK docs fixed (ODK internal, pull request), but much of ODK's docs has moved to OBOOK (OBO Academy)
- Fixing some issues with default values when using
robot_report
in your ODK config (pull request)
New commands:
sh run.sh make help
: show frequently used ODK commands and their usage (issue)sh run.sh make TSV=my.tsv validate-tsv
: Experimental TSV validation feature with TSValid (issue)sh run.sh make validate-all-tsv
: Validate all TSV tables configured in repo with TSValid (issue)sh run.sh make dosdp-matches-%
: Run the "matches" pipeline, see above. (issue)sh run.sh make clean
: Clean up some temporary files created by the build. (issue)
Recommendations for ontology maintainers:
- We have overhauled the SPARQL query management to make it more customisable (pull request). We recommend, before updating to the latest ODK, to delete old and potentially stale SPARQL queries like this:
rm ../sparql/owldef-violation.sparql ../sparql/nolabels-violation.sparql ../sparql/def-lacks-xref-violation.sparql ../sparql/obsolete-violation.sparql ../sparql/redundant-subClassOf-violation.sparql
. - You should periodically review the contents of your
src/ontology/custom.Makefile
- if you overwrite a lot of goals, you should review wether the overwrite is still necessary. This ODK release has a lot of revisions to the Makefile, so this should be a good opportunity to check this!
December 2021 Release 2
With ROBOT 1.8.3, which patches an issue with ROBOT report (custom queries) and also gets rid of Log4J.
No changes to ODK repo, so no need to run updates. To update your ODK container:
docker pull obolibrary/odkfull
December 2021 release
Attention: breaking changes below!
- Updated to ROBOT 1.8.2
- Updated to DOSDP tools 0.18
- Adding
use_base_merging
to config files, which enables the BASE file pipeline, a completely different way to handle imports. This feature is not ready for primetime, but if you are interested in testing this, get in touch. For more details read (pull). - Adding back ssh/scp
- Migrated to Java 11 as the base Java in ODK (pull)
- Adding
make_base
feature that allows to autogenerate base files from ontologies where they do not exist (pull) - Adding new command
sh run.sh make no-mirror-refresh-imports
which refreshes imports without refreshing mirrors. Can be used for individual ontologies as well. - Making
owltools
where necessary configurable with a bespoke memory parameter (pull) - Fixing the GitHub action to auto-deploy the documentation (pr).
- Fixed a bug where the DOSDP pages in ODK where generated in the wrong part of the mkdocs documentation.
- New command
sh run.sh make explain_unsat
which generates a nicely formatted set of explanations for your unsatisfiable classes (pull) - Adding method to measure the memory consumption of your builds. For example, you can now run
IMAGE=odklite ODK_DEBUG=yes ./run.sh make prepare_release
to run your release on the (much lighter)odklite
container of ODK, and get a nice benchmark summary at the end:
### DEBUG STATS ###
Elapsed time: 7:49.24
Peak memory: 6517356 kb
- Breaking changes:
- Imported modules are no longer generated in OBO format automatically (only OWL). This reflects the fact that in most ontologies, even those where the editors file is in OBO format, imported ontologies are usually in OWL format. You can use the
export_obo
option to add them back (pull) - Equivalent class default setting changed from
all
toasserted only
(pull). This means that from now on, if you dont change the setting deliberately, your pipeline will fail if their are equivalent classes that are not deliberately asserted. - OWL 2 DL profile checking is now
true
by default. You have to actively switch it off by settingensure_owl2dl_profile
toFALSE
in your config file.
- Imported modules are no longer generated in OBO format automatically (only OWL). This reflects the fact that in most ontologies, even those where the editors file is in OBO format, imported ontologies are usually in OWL format. You can use the
To update to the latest version of ODK:
docker pull obolibrary/odkfull
To mirgate your repository to the latest version of ODK:
cd myont/src/ontology
sh run.sh make update_repo
sh run.sh make update_repo #(needs to be run twice since the update process also updates the update process.)
October 2021 release
- Important: The way we install python packages has changed significantly: we are now using fixed version dependencies (issue). If there are problems with the versions of packages we are using, please let us know immediately.
- We are now using multi-stage builds in ODK for docker. The
obolibrary/odklite
container is considerably lighter (smaller) than the normalodkfull
container you have been using so far, and it should be sufficient for most ODK pipelines. - Mirrors are now downloaded with CURL instead of ROBOT, and configurable with
mirror_retry_download
andmirror_max_time_download
in theimport_group
section of the ODK config file (PR). Note that the defaultmirror_max_time_download
is 200 sec (a bit more than 3 minutes), which may be tight for some huge ontologies. - odkfull now includes Soufflé.
- For those who are using DOSDP patterns we re-introduced the pattern schema check even if pattern generation is skipped (it is very fast).
- Created a simple way to cite ODK from within GitHub, using the CITATION.cff file. If you go to https://github.com/INCATools/ontology-development-kit, you will now see a "cite this repo" section under the About section.
- Changed the default README.md to include a better reference to ODK, the correct ontology description text and the correct edit file extension.
- Added a GitHub action to deploy the ODK-based mkdocs documentation (issue).
- Created a page for frequently used ODK commands.
- Bugfixes:
- project repo name was not read correctly during
make update_repo
causing it to be named "False" (commit) - Fixed a bug where mirrors were not considered precious
- Fixed a bug where where the report directories where not created when running a sparql report command (commit)
- Fixed a bug where
report
command incorrectly did not include components in the check (issue) - Fixed a bug where myont-odk.yaml was not created when using command line mode during seeding (issue)
- Added some tests for external tools (issue) to ODK built process
- Konclude now works on the arm64 variant.
- project repo name was not read correctly during