Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
fix(excel2resources, excel2properties): cover all cases (DEV-1040) (#201
)

* fix bugs, make code more robust
* PR-title: allow commas and numbers in scope
* fix DeprecationWarning in JSON Schema validation
* allow multiple superclasses for properties, add column 'gui_attributes'
* cover more cases in the test data
* add Romansh
  • Loading branch information
jnussbaum committed Jun 23, 2022
1 parent 5c56601 commit 4c6ed19
Show file tree
Hide file tree
Showing 23 changed files with 501 additions and 287 deletions.
Expand Up @@ -12,7 +12,7 @@ jobs:
# check PR title
- uses: deepakputhraya/action-pr-title@master
with:
regex: '([a-z])+(\(([a-z\-_ ])+\))?!?: [a-z]([a-zA-Z-\.\d \(\)\[\]#_,])+$' # Regex the title should match.
regex: '([a-z])+(\(([0-9a-z\-_, ])+\))?!?: [a-z]([a-zA-Z-\.\d \(\)\[\]#_,])+$' # Regex the title should match.
allowed_prefixes: "fix,refactor,feat,docs,chore,style,test" # title should start with the given prefix
disallowed_prefixes: "feature,hotfix" # title should not start with the given prefix
prefix_case_sensitive: true # title prefix are case insensitive
Expand Down
1 change: 0 additions & 1 deletion MANIFEST.in
Expand Up @@ -4,4 +4,3 @@ include knora/dsplib/schemas/lists-only.json
include knora/dsplib/schemas/resources-only.json
include knora/dsplib/schemas/properties-only.json
include knora/dsplib/schemas/data.xsd
include knora/dsplib/utils/language-codes-3b2_csv.csv
2 changes: 1 addition & 1 deletion Pipfile
Expand Up @@ -17,6 +17,7 @@ pystrict = "*"
openpyxl = "*"
pyparsing = "==2.4.7"
networkx = "*"
pandas = "*"

[dev-packages]
mkdocs = "*"
Expand All @@ -28,7 +29,6 @@ setuptools = "*"
wheel = "*"
pipenv-setup = "*"
pytest = "*"
pandas = "*"

[requires]
python_version = "3.9"
150 changes: 79 additions & 71 deletions Pipfile.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 1 addition & 4 deletions dev-requirements.txt
Expand Up @@ -23,10 +23,8 @@ mkdocs-material~=8.3.6
mkdocs-material-extensions~=1.0.3
mypy~=0.961
mypy-extensions~=0.4.3
numpy~=1.22.4
orderedmultidict~=1.0.1
packaging~=20.9
pandas~=1.4.2
pep517~=0.12.0
pip~=22.1.2
pip-shims~=0.7.0
Expand All @@ -42,12 +40,11 @@ pymdown-extensions~=9.5
pyparsing~=2.4.7
pytest~=7.1.2
python-dateutil~=2.8.2
pytz~=2022.1
pyyaml~=6.0
pyyaml-env-tag~=0.1
requests~=2.28.0
requirementslib~=1.6.4
setuptools~=62.4.0
setuptools~=62.6.0
six~=1.16.0
toml~=0.10.2
tomli~=2.0.1
Expand Down
Binary file modified docs/assets/images/img-properties-example.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/assets/images/img-resources-example-1.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/assets/templates/properties_template.xlsx
Binary file not shown.
Binary file modified docs/assets/templates/resources_template.xlsx
Binary file not shown.
35 changes: 19 additions & 16 deletions docs/dsp-tools-excel.md
Expand Up @@ -20,25 +20,25 @@ The expected worksheets of the Excel file are:
- `classes`: a table with all resource classes intended to be used in the resulting JSON
- `class1`, `class2`,...: a table for each resource class named after its name

The Excel sheet must have the following structure.

The worksheet called `classes` has the following structure:
The worksheet called `classes` must have the following structure:
![img-resources-example-1.png](assets/images/img-resources-example-1.png)

The expected columns are:

- `name` : The name of the resource
- `en`, `de`, `fr`, `it` : The labels of the resource in different languages, at least one language has to be provided
- `comment_en`, `comment_de`, `comment_fr`, `comment_it`: optional comments in the respective language
- `super` : The base class of the resource
- `name` (mandatory): The name of the resource
- `en`, `de`, `fr`, `it`, `rm`: The labels of the resource in different languages, at least one language has to be provided
- `comment_en`, `comment_de`, `comment_fr`, `comment_it`, `comment_rm` (optional): comments in the respective language
- `super` (mandatory): The base class(es) of the resource, separated by commas

The optional columns may be omitted in the Excel.

All other worksheets, one for each resource class, have the following structure:
![img-resources-example-2.png](assets/images/img-resources-example-2.png){ width=50% }

The expected columns are:

- `Property` : The name of the property
- `Cardinality` : The cardinality, one of: `1`, `0-1`, `1-n`, `0-n`
- `Property` (mandatory): The name of the property
- `Cardinality` (mandatory): The cardinality, one of: `1`, `0-1`, `1-n`, `0-n`

The GUI order is given by the order in which the properties are listed in the Excel sheet.

Expand All @@ -58,15 +58,18 @@ The Excel sheet must have the following structure:

The expected columns are:

- `name` : The name of the property
- `super` : The base property of the property
- `object` : If the property is derived from `hasValue`, the type of the property must be further specified by the
- `name` (mandatory): The name of the property
- `super` (mandatory): The base property/ies of the property, separated by commas
- `object` (mandatory): If the property is derived from `hasValue`, the type of the property must be further specified by the
object it takes, e.g. `TextValue`, `ListValue`, or `IntValue`. If the property is derived from `hasLinkTo`,
the `object` specifies the resource class that this property refers to.
- `en`, `de`, `fr`, `it` : The labels of the property in different languages, at least one language has to be provided
- `comment_en`, `comment_de`, `comment_fr`, `comment_it`: optional comments in the respective language
- `gui_element` : The GUI element for the property
- `hlist` : In case of list values: the name of the list
- `en`, `de`, `fr`, `it`, `rm`: The labels of the property in different languages, at least one language has to be provided
- `comment_en`, `comment_de`, `comment_fr`, `comment_it`, `comment_rm` (optional): comments in the respective language
- `gui_element` (mandatory): The GUI element for the property
- `gui_attributes` (optional): The gui_attributes in the form "attr: value, attr: value".

The optional columns may be omitted in the Excel.
For backwards compatibility, files containing a column `hlist` are valid, but deprecated.

For further information about properties, see [here](./dsp-tools-create-ontologies.md#properties).

Expand Down
2 changes: 1 addition & 1 deletion knora/dsplib/schemas/lists-only.json
@@ -1,5 +1,5 @@
{
"$schema": "https://json-schema.org/draft-07/schema",
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://dasch.swiss/schema/lists.json",
"title": "JSON schema for DSP lists",
"description": "JSON schema for the lists section used in DSP ontologies",
Expand Down
2 changes: 1 addition & 1 deletion knora/dsplib/schemas/ontology.json
@@ -1,5 +1,5 @@
{
"$schema": "https://json-schema.org/draft-07/schema",
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://dasch.swiss/schema/ontology.json",
"title": "JSON schema for DSP ontologies",
"description": "JSON schema for DSP ontologies",
Expand Down
3 changes: 2 additions & 1 deletion knora/dsplib/schemas/properties-only.json
@@ -1,5 +1,5 @@
{
"$schema": "https://json-schema.org/draft-07/schema",
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://dasch.swiss/schema/properties-only.json",
"title": "JSON schema for properties used in DSP ontologies",
"description": "JSON schema for the properties section used in DSP ontologies",
Expand Down Expand Up @@ -82,6 +82,7 @@
"ListValue",
"Region",
"Resource",
"Representation",
"Annotation"
]
},
Expand Down
2 changes: 1 addition & 1 deletion knora/dsplib/schemas/resources-only.json
@@ -1,5 +1,5 @@
{
"$schema": "https://json-schema.org/draft-07/schema",
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://dasch.swiss/schema/resources-only.json",
"title": "JSON schema for resources used in DSP ontologies",
"description": "JSON schema for the resources section used in DSP ontologies",
Expand Down

0 comments on commit 4c6ed19

Please sign in to comment.