Skip to content

Commit

Permalink
style(code-style): reformat files (DSP-1906) (#96)
Browse files Browse the repository at this point in the history
* add python style configuration

* remove .py section in .editorconfig

* reformat .md files

* Squashed commit of the following:

commit 00d3545
Author: irinaschubert <irina.schubert@dasch.swiss>
Date:   Thu Sep 16 15:01:16 2021 +0200

    update documentation for markdown

* use autopep8 on *.py files

* reformat *.md files

* reformat files

* update schemas

* Update excel_to_json_properties.py

* fix file paths in Manifest

* update requirements.txt
  • Loading branch information
irinaschubert committed Sep 20, 2021
1 parent 028d208 commit 18964d1
Show file tree
Hide file tree
Showing 53 changed files with 1,945 additions and 1,647 deletions.
3 changes: 2 additions & 1 deletion .github/pull_request_template.md
@@ -1,6 +1,7 @@
===REMOVE===

Important! Please follow the new guidelines for naming Pull Requests: https://docs.dasch.swiss/developers/dsp/contribution/#pull-request-guidelines
Important! Please follow the new guidelines for naming Pull
Requests: https://docs.dasch.swiss/developers/dsp/contribution/#pull-request-guidelines

> **Note:** If a pull request consists consists of *only one* commit when squash-merging it to main, the commit message will *not* be correct. In this case you have to manually ensure that the commit message is identical to the PR title, not the commit that the PR consists of.
Expand Down
145 changes: 85 additions & 60 deletions CHANGELOG.md

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions MANIFEST.in
@@ -1,8 +1,8 @@
include README.md
include knora/dsplib/utils/knora-schema.json
include knora/dsplib/utils/knora-schema-lists.json
include knora/dsplib/utils/knora-schema-lists-only.json
include knora/dsplib/utils/knora-schema-resources-only.json
include knora/dsplib/utils/knora-schema-properties-only.json
include knora/dsplib/utils/knora-data-schema.xsd
include knora/dsplib/schemas/ontology.json
include knora/dsplib/schemas/lists.json
include knora/dsplib/schemas/lists-only.json
include knora/dsplib/schemas/resources-only.json
include knora/dsplib/schemas/properties-only.json
include knora/dsplib/schemas/data.xsd
include knora/dsplib/utils/language-codes-3b2_csv.csv
367 changes: 196 additions & 171 deletions docs/dsp-tools-create.md

Large diffs are not rendered by default.

55 changes: 28 additions & 27 deletions docs/dsp-tools-excel.md
Expand Up @@ -2,15 +2,15 @@

# Excel files for data modelling and data import

dsp-tools is able to process Excel files and output the appropriate JSON or XML file. The JSON/XML file can then be used to create
the ontology on the DSP server or import data to the DSP repository. dsp-tools can also be used to create a list from an Excel
file.
dsp-tools is able to process Excel files and output the appropriate JSON or XML file. The JSON/XML file can then be used
to create the ontology on the DSP server or import data to the DSP repository. dsp-tools can also be used to create a
list from an Excel file.

## Create the resources for a data model from an Excel file

With dsp-tools the `resources` section used in a data model (JSON) can be created from an Excel file. Only the first worksheet of
the Excel file is considered and only `XLSX` files are allowed. The `resources` section can be inserted into the ontology file and
then be uploaded onto a DSP server.
With dsp-tools the `resources` section used in a data model (JSON) can be created from an Excel file. Only the first
worksheet of the Excel file is considered and only `XLSX` files are allowed. The `resources` section can be inserted
into the ontology file and then be uploaded onto a DSP server.

The Excel sheet must have the following format:
![img-resources-example.png](assets/images/img-resources-example.png)
Expand Down Expand Up @@ -49,39 +49,40 @@ For further information about properties, see [here](./dsp-tools-create.md#prope

## Create a list from one or several Excel files

With dsp-tools a JSON list can be created from one or several Excel files. The list can then be inserted into a JSON ontology and
uploaded to a DSP server. The expected format of the Excel files is described [here](./dsp-tools-create.md#lists-from-excel). It
is possible to create multilingual lists. In this case, a separate Excel file has to be created for each language. The data has to
be in the first worksheet of the Excel file(s). It is important that all the Excel lists have the same structure. So, the
translation(s) of a label in one Excel sheet has to be in the exact same cell (i.e. with the same cell index) in its own Excel
sheet.
With dsp-tools a JSON list can be created from one or several Excel files. The list can then be inserted into a JSON
ontology and uploaded to a DSP server. The expected format of the Excel files is
described [here](./dsp-tools-create.md#lists-from-excel). It is possible to create multilingual lists. In this case, a
separate Excel file has to be created for each language. The data has to be in the first worksheet of the Excel file(s).
It is important that all the Excel lists have the same structure. So, the translation(s) of a label in one Excel sheet
has to be in the exact same cell (i.e. with the same cell index) in its own Excel sheet.

Only Excel files with file extension `.xlsx` are considered. All Excel files have to be located in the same directory. When
calling the `excel` command, this folder is provided as an argument to the call. The language of the labels has to be provided in
the Excel file's file name after an underline and before the file extension, p.ex. `liste_de.xlsx` would be considered a list with
German (`de`) labels, `list_en.xlsx` a list with English (`en`) labels. The language has to be a valid ISO 639-1 or ISO 639-2
language code.
Only Excel files with file extension `.xlsx` are considered. All Excel files have to be located in the same directory.
When calling the `excel` command, this folder is provided as an argument to the call. The language of the labels has to
be provided in the Excel file's file name after an underline and before the file extension, p.ex. `liste_de.xlsx` would
be considered a list with German (`de`) labels, `list_en.xlsx` a list with English (`en`) labels. The language has to be
a valid ISO 639-1 or ISO 639-2 language code.

The following example shows how to create a JSON list from two Excel files which are in a directory called `lists`. The output is
written to the file `list.json`.
The following example shows how to create a JSON list from two Excel files which are in a directory called `lists`. The
output is written to the file `list.json`.

```bash
dsp-tools excel lists list.json
```

The two Excel files `liste_de.xlsx` and `list_en.xlsx` are located in a folder called `lists`. `liste_de.xlsx` contains German
labels for the list, `list_en.xlsx` contains the English labels.
The two Excel files `liste_de.xlsx` and `list_en.xlsx` are located in a folder called `lists`. `liste_de.xlsx` contains
German labels for the list, `list_en.xlsx` contains the English labels.

```
lists
|__ liste_de.xlsx
|__ list_en.xlsx
```

For each list node, the `label`s are read from the Excel files. The language code, provided in the file name, is then used for the
labels. As node `name`, a simplified version of the English label is taken if English is one of the available languages. If
English is not available, one of the other languages is chosen (which one depends on the representation of the file order). If
there are two node names with the same name, an incrementing number is appended to the `name`.
For each list node, the `label`s are read from the Excel files. The language code, provided in the file name, is then
used for the labels. As node `name`, a simplified version of the English label is taken if English is one of the
available languages. If English is not available, one of the other languages is chosen (which one depends on the
representation of the file order). If there are two node names with the same name, an incrementing number is appended to
the `name`.

```JSON
{
Expand Down Expand Up @@ -116,5 +117,5 @@ there are two node names with the same name, an incrementing number is appended
}, ...
```

After the creation of the list, a validation against the JSON schema for lists is performed. An error message ist printed out if
the list is not valid. Furthermore, it is checked that no two nodes are the same.
After the creation of the list, a validation against the JSON schema for lists is performed. An error message ist
printed out if the list is not valid. Furthermore, it is checked that no two nodes are the same.
39 changes: 21 additions & 18 deletions docs/dsp-tools-usage.md
Expand Up @@ -30,12 +30,12 @@ The following options are available:
- `-u` | `--user` _username_: username used for authentication with the DSP API (default: root@example.com)
- `-p` | `--password` _password_: password used for authentication with the DSP API (default: test)
- `-V` | `--validate`: If set, only the validation of the JSON file is performed.
- `-l` | `--lists`: If set, only the lists are created using a [simplified schema](./dsp-tools-create.md#lists). Please note that
in this case the project must already exist.
- `-l` | `--lists`: If set, only the lists are created using a [simplified schema](./dsp-tools-create.md#lists). Please
note that in this case the project must already exist.
- `-v` | `--verbose`: If set, some information about the progress is printed to the console.

The command is used to read the definition of a data model (provided in a JSON file) and create it on the DSP server. The
following example shows how to load the ontology defined in `data_model_definition.json` onto the DSP
The command is used to read the definition of a data model (provided in a JSON file) and create it on the DSP server.
The following example shows how to load the ontology defined in `data_model_definition.json` onto the DSP
server `https://api.dsl.server.org` provided with the `-s` option. The username `root@example.com` and the password
`test` are used.

Expand All @@ -60,10 +60,10 @@ The following options are available:
[IRI](https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier) of the project
- `-v` | `--verbose`: If set, some information about the progress is printed to the console.

The command is used to get the definition of a data model from a DSP server and write it into a JSON file. This JSON file could
then be used to upload the data model to another DSP server. The following example shows how to get the data model from a DSP
server `https://api.dsl.server.org` provided with the `-s` option. The username `root@example.com` and the password `test` are
used. The data model is saved into the output file `output_file.json`.
The command is used to get the definition of a data model from a DSP server and write it into a JSON file. This JSON
file could then be used to upload the data model to another DSP server. The following example shows how to get the data
model from a DSP server `https://api.dsl.server.org` provided with the `-s` option. The username `root@example.com` and
the password `test` are used. The data model is saved into the output file `output_file.json`.

```bash
dsp-tools get -s https://api.dsl.server.org -u root@example.com -p test output_file.json
Expand All @@ -84,9 +84,10 @@ The following options are available:
- `-S` | `--sipi` _SIPIserver_: URL of the SIPI IIIF server (default: http://0.0.0.0:1024)
- `-v` | `--verbose`: If set, more information about the uploaded resources is printed to the console.

The command is used to upload data defined in an XML file onto a DSP server. The following example shows how to upload data from
an XML file `xml_data_file.xml` onto the DSP server `https://api.dsl.server.org` provided with the `-s` option. The
username `root@example.com` and the password `test` are used. The interface for the SIPI IIIF server is provided with the `-S`
The command is used to upload data defined in an XML file onto a DSP server. The following example shows how to upload
data from an XML file `xml_data_file.xml` onto the DSP server `https://api.dsl.server.org` provided with the `-s`
option. The username `root@example.com` and the password `test` are used. The interface for the SIPI IIIF server is
provided with the `-S`
option (`https://iiif.dsl.server.org`).

```bash
Expand All @@ -103,21 +104,23 @@ dsp-tools excel [option] folder_with_excel_files output_file.json

The following option is available:

- `-l` | `--listname` _listname_: name to be used for the list (filename before last occurrence of `_` is used if omitted)
- `-l` | `--listname` _listname_: name to be used for the list (filename before last occurrence of `_` is used if
omitted)

The command is used to create a JSON list file from one or several Excel files. It is possible to create multilingual lists.
Therefore, an Excel file for each language has to be provided. The data has to be in the first worksheet of the Excel file and all
Excel files have to be in the same directory. When calling the `excel` command, this directory has to be provided as an argument
to the call.
The command is used to create a JSON list file from one or several Excel files. It is possible to create multilingual
lists. Therefore, an Excel file for each language has to be provided. The data has to be in the first worksheet of the
Excel file and all Excel files have to be in the same directory. When calling the `excel` command, this directory has to
be provided as an argument to the call.

The following example shows how to create a JSON list from Excel files in a directory called `lists`.

```bash
dsp-tools excel lists list.json
```

The description of the expected Excel format can be found [here](./dsp-tools-create.md#lists-from-excel). More information about
the usage of this command can be found [here](./dsp-tools-excel.md#create-a-list-from-one-or-several-excel-files).
The description of the expected Excel format can be found [here](./dsp-tools-create.md#lists-from-excel). More
information about the usage of this command can be
found [here](./dsp-tools-excel.md#create-a-list-from-one-or-several-excel-files).

## Create resources from an Excel file

Expand Down

0 comments on commit 18964d1

Please sign in to comment.