Skip to content

Commit

Permalink
feat(api-v1): Change API v1 file uploads to work like API v2 (DSP-41,…
Browse files Browse the repository at this point in the history 
… PR 3) (#1722)
  • Loading branch information
Benjamin Geer committed Oct 6, 2020
1 parent 68e67f8 commit a824bcc
Show file tree
Hide file tree
Showing 58 changed files with 1,105 additions and 3,118 deletions.
160 changes: 72 additions & 88 deletions docs/03-apis/api-v1/adding-resources.md
View file
Expand Up @@ -47,67 +47,53 @@ The request header's content type has to be set to `application/json`.

## Adding Resources with Image Files

Certain resource classes can have attached image files. There are two ways to
attach a file to a resource: Either by submitting directly the binaries of the file in a
an HTTP Multipart request, or by indicating the location of the file. The two cases are referred to
as non-GUI case and GUI case (see [Sipi and Knora](../../07-sipi/sipi-and-knora.md)).

### Including the binaries (non-GUI case)

In order to include the binaries, a HTTP Multipart request has to be
sent. One part contains the JSON (same format as described for
[Adding Resources Without Images Files](#adding-resources-without-a-digital-representation))
and has to be named `json`. The other part contains the file's name, its binaries, and its mime type
and has to be named `file`. The following example illustrates how to
make this type of request using Python 3:

```python
#!/usr/bin/env python3

import requests, json

# a Python dictionary that will be turned into a JSON object
resourceParams = {
'restype_id': 'http://www.knora.org/ontology/test#testType',
'properties': {
'http://www.knora.org/ontology/test#testtext': [
{'richtext_value': {'utf8str': "test"}}
],
'http://www.knora.org/ontology/test#testnumber': [
{'int_value': 1}
]
},
'label': "test resource",
'project_id': 'http://rdfh.ch/projects/testproject'
}

# the name of the file to be submitted
filename = "myimage.jpg"
The first step is to upload an image file to Sipi, using a
`multipart/form-data` request, where `sipihost` represents the host and
port on which Sipi is running:

# a tuple containing the file's name, its binaries and its mimetype
file = {'file': (filename, open(filename, 'rb'), "image/jpeg")} # use name "file"
```
HTTP POST to http://sipihost/upload?token=TOKEN
```

# do a POST request providing both the JSON and the binaries
r = requests.post("http://host/v1/resources",
data={'json': json.dumps(resourceParams)}, # use name "json"
files=file,
auth=('user', 'password'))
The `TOKEN` is the `sid` returned by Knora in response to the
client's login request (see [Authentication](authentication.md)).
The request must contain a body part providing the file as well as a parameter
`filename`, providing the file's original filename, which both Knora and Sipi will
store; these filenames can be descriptive and need not be unique.

Sipi will then convert the uploaded image file to JPEG 2000 format and store
it in a temporary location. If this is successful, it will return a JSON
response that looks something like this:

```json
{
"uploadedFiles": [{
"originalFilename": "manuscript-1234-page-1.tiff",
"internalFilename": "3UIsXH9bP0j-BV0D4sN51Xz.jp2",
"temporaryBaseIIIFUrl": "http://sipihost/tmp"
}]
}
```

Please note that the file has to be read in binary mode (by default it
would be read in text mode).
This provides:

### Indicating the location of a file (GUI case)
- the `originalFilename`, which we submitted when uploading the file
- the unique `internalFilename` that Sipi has randomly generated for the file
- the `temporaryBaseIIIFUrl`, which we can use to construct a IIIF URL for
previewing the file

This request works similarly to
[Adding Resources Without Image Files](#adding-resources-without-a-digital-representation). The JSON format is described
in the TypeScript interface `createResourceWithRepresentationRequest` in
module `createResourceFormats`. The request header's content type has to
set to `application/json`.
The client may now wish to get a thumbnail of the uploaded image, to allow
the user to confirm that the correct files have been uploaded. This can be done
by adding the filename and IIIF parameters to `temporaryBaseIIIFUrl`. For example, to get
a JPG thumbnail image whose width and height are at most 128 pixels wide, you would request
`http://sipihost/tmp/3UIsXH9bP0j-BV0D4sN51Xz.jp2/full/!128,128/0/default.jpg`.

In addition to [Adding Resources Without Image Files](#adding-resources-without-a-digital-representation), the
(temporary) name of the file, its original name, and mime type have to
be provided (see [GUI Case](../../07-sipi/sipi-and-knora.md#gui-case)).
The request to Knora works similarly to
[Adding Resources Without Image Files](#adding-resources-without-image-files),
with the addition of `file`, whose value is the `internalFilename` that Sipi returned.
See the TypeScript interface `createResourceWithRepresentationRequest` in
module `createResourceFormats` for details. The request header's content type must be
set to `application/json`.

## Response to a Resource Creation

Expand All @@ -122,7 +108,7 @@ The JSON format of the response is described in the TypeScript interface
## Changing a Resource's Label

A resource's label can be changed by making a PUT request to the path
segments `resources/label`. The resource's Iri has to be provided in the
segments `resources/label`. The resource's IRI has to be provided in the
URL (as its last segment). The new label has to submitted as JSON in the
HTTP request's body.

Expand Down Expand Up @@ -150,18 +136,24 @@ Only system or project administrators may use the bulk import.
The procedure for using this feature is as follows
(see the [example below](#bulk-import-example)).

1. Make an HTTP GET request to Knora to [get XML schemas](#1-get-xml-schemas) describing
the XML to be provided for the import.
2. [Generate an XML import document](#2-generate-xml-import-document) representing the
data to be imported, following the Knora import schemas that were generated in step 1.
You will probably want to write a script to do this. Knora is not involved in this step.
If you are also importing image files, this XML document needs to
[contain the filesystem paths](#bulk-import-with-image-files) of those files.
3. [Validate your XML import document](#3-validate-xml-import-document), using an XML schema validator such as
[Apache Xerces](http://xerces.apache.org) or [Saxon](http://www.saxonica.com), or an
XML development environment such as [Oxygen](https://www.oxygenxml.com). This will
help ensure that the data you submit to Knora is correct. Knora is not involved in this step.
4. [Submit the XML import document to Knora](#4-submit-xml-import-document-to-knora).
1. Make an HTTP GET request to Knora to [get XML schemas](#1-get-xml-schemas) describing
the XML to be provided for the import.

2. If you are importing image files, [upload files to Sipi](#2-upload-files-to-sipi).

3. [Generate an XML import document](#3-generate-xml-import-document) representing the
data to be imported, following the Knora import schemas that were generated in step 1.
You will probably want to write a script to do this. Knora is not involved in this step.
If you are also importing image files, this XML document needs to
[contain the filenames](#bulk-import-with-image-files) that Sipi returned
for the files you uploaded in step 2.

4. [Validate your XML import document](#4-validate-xml-import-document), using an XML schema validator such as
[Apache Xerces](http://xerces.apache.org) or [Saxon](http://www.saxonica.com), or an
XML development environment such as [Oxygen](https://www.oxygenxml.com). This will
help ensure that the data you submit to Knora is correct. Knora is not involved in this step.

5. [Submit the XML import document to Knora](#5-submit-xml-import-document-to-knora).

In this procedure, the person responsible for generating the XML import
data need not be familiar with RDF or with the ontologies involved.
Expand Down Expand Up @@ -206,7 +198,12 @@ containing three files:

- `knoraXmlImport.xsd`: The standard Knora XML import schema, used by all XML imports.

#### 2. Generate XML Import Document
#### 2. Upload Files to Sipi

See [Upload Files to Sipi](../api-v2/editing-values.md#upload-files-to-sipi) in
the Knora API v2 documentation.

#### 3. Generate XML Import Document

We now convert our existing data to XML, probably by writing a custom
script. The resulting XML import document could look like this:
Expand Down Expand Up @@ -305,18 +302,6 @@ This illustrates several aspects of XML imports:

- The type of each value must be specified using the attribute
`knoraType`.

- If a property has `knoraType="date_value"`, the date value must have the following format:

```
(GREGORIAN|JULIAN|ISLAMIC):\d{1,4}(-\d{1,2}(-\d{1,2})?)?( BC| AD| BCE| CE)?(:\d{1,4}(-\d{1,2}(-\d{1,2})?)?( BC| AD| BCE| CE)?)?
```

E.g. an exact date like `GREGORIAN:2015-12-03` or a period like `GREGORIAN:2015-12-03:2015-12-04`.
Dates may also have month or year precision, e.g. `ISLAMIC:1407-02` (the whole month of december) or `JULIAN:1330`
(the whole year 1330). An optional ERA indicator term (`BCE`, `CE`, or `BC`, `AD`) can be added to the date, when no
era is provided the default era `AD` will be considered. Era can be given as `GREGORIAN:1220 BC` or in range as
`GREGORIAN:600 BC:480 BC`.

- A link to another resource described in the XML import is
represented as a child element of a property element, with
Expand Down Expand Up @@ -351,7 +336,7 @@ This illustrates several aspects of XML imports:
- A text value can have a `lang` attribute, whose value is an ISO 639-1
code specifying the language of the text.

#### 3. Validate XML Import Document
#### 4. Validate XML Import Document

You can use an XML schema validator such as [Apache Xerces](http://xerces.apache.org) or
[Saxon](http://saxon.sourceforge.net/), or an XML development environment
Expand All @@ -364,7 +349,7 @@ For example, using Saxon:
java -cp ./saxon9ee.jar com.saxonica.Validate -xsd:p0801-biblio.xsd -s:data.xml
```

#### 4. Submit XML Import Document to Knora
#### 5. Submit XML Import Document to Knora

To create these resources in Knora, make an HTTP post request with the XML import document
as the request body. The URL must specify the (URL-encoded) IRI of the project in which
Expand Down Expand Up @@ -421,8 +406,8 @@ contains the IRI of the target resource.

To attach an image file to a resource, we must provide the
element `knoraXmlImport:file` before the property elements. In this
element, we must give the absolute filesystem path to the file that
should be attached to the resource, along with its MIME type:
element, we must provide a `filename` attribute, containing the `internalFilename`
that Sipi returned for the file in [2. Upload Files to Sipi](#2-upload-files-to-sipi).

```xml
<?xml version="1.0" encoding="UTF-8"?>
Expand All @@ -437,7 +422,7 @@ should be attached to the resource, along with its MIME type:
</incunabula:book>
<incunabula:page id="test_page">
<knoraXmlImport:label>a page with an image</knoraXmlImport:label>
<knoraXmlImport:file path="/usr/local/share/import-images/incunabula/12345.tiff" mimetype="image/tiff"/>
<knoraXmlImport:file filename="67SEfNU1wK2-CSf5abe2eh3.jp2"/>
<incunabula:origname knoraType="richtext_value">Chlaus</incunabula:origname>
<incunabula:pagenum knoraType="richtext_value">1a</incunabula:pagenum>
<incunabula:partOf>
Expand All @@ -448,6 +433,5 @@ should be attached to the resource, along with its MIME type:
</knoraXmlImport:resources>
```

During the processing of the bulk import, Knora will
communicate the location of file to Sipi, which will convert it to JPEG 2000
for storage.
During the processing of the bulk import, Knora will ask Sipi for the rest of the
file's metadata, and store that metadata in a file value attached to the resource.
51 changes: 10 additions & 41 deletions docs/03-apis/api-v1/changing-values.md
View file
Expand Up @@ -56,52 +56,21 @@ has to be used in order to create a new value (all these TypeScript interfaces a

## Modifying a File Value

In order to exchange a file value (digital representation of a
resource), the path segment `filevalue` has to be used. The IRI of the
resource whose file value is to be exchanged has to be appended:
To change a file value, the client first uploads the new file to
Sipi, following the procedure described in
[Adding Resources with Image Files](adding-resources.md#adding-resources-with-image-files).

Then the client sends a request to Knora, using this following route:

```
HTTP PUT to http://host/filevalue/resourceIRI
```

Please note that the resource IRI has to be URL encoded.

There are two ways to change a file of a resource: Either by submitting
directly the binaries of the file in a HTTP Multipart request or by
indicating the location of the file. The two cases are referred to as
non-GUI case and GUI case (TODO: add a link to "Sipi and Knora").

### Including the binaries (non-GUI case)

Here, a HTTP MULTIPART request has to be made simply providing the
binaries (without JSON):

```python
#!/usr/bin/env python3

import requests, json, urllib

# the name of the file to be submitted
filename = 'myimage.tif'

# a tuple containing the file's name, its binaries and its mimetype
files = {'file': (filename, open(filename, 'rb'), "image/tiff")}

resIri = urllib.parse.quote_plus('http://rdfh.ch/xy')

r = requests.put("http://host/filevalue/" + resIri,
files=files)
```

Please note that the file has to be read in binary mode (by default it
would be read in text mode).

### Indicating the location of a file (GUI case)

Here, simply the location of the new file has to be submitted as JSON.
The JSON format is described in the TypeScript interface
`changeFileValueRequest` in module `changeValueFormats`. The request
header's content type has to set to `application/json`.
Here, `resourceIRI` is the URL-encoded IRI of the resource whose file value is
to be changed. The body of the request is a JSON object described in the TypeScript
interface `changeFileValueRequest` in module `changeValueFormats`, and contains
`file`, whose value is the `internalFilename` that Sipi returned. The request header's
content type must be set to `application/json`.

## Response on Value Change

Expand Down
4 changes: 2 additions & 2 deletions docs/03-apis/api-v2/editing-values.md
View file
Expand Up @@ -262,11 +262,11 @@ Sipi then returns a JSON response that looks something like this:
"uploadedFiles": [{
"originalFilename": "manuscript-1234-page-1.tiff",
"internalFilename": "3UIsXH9bP0j-BV0D4sN51Xz.jp2",
"temporaryBaseIIIFUrl": "http://sipihost/tmp/3UIsXH9bP0j-BV0D4sN51Xz.jp2"
"temporaryBaseIIIFUrl": "http://sipihost/tmp"
}, {
"originalFilename": "manuscript-1234-page-2.tiff",
"internalFilename": "2RvJgguglpe-B45EOk0Gx8H.jp2",
"temporaryBaseIIIFUrl": "http://sipihost/tmp/2RvJgguglpe-B45EOk0Gx8H.jp2"
"temporaryBaseIIIFUrl": "http://sipihost/tmp"
}]
}
```
Expand Down
4 changes: 2 additions & 2 deletions docs/05-internals/design/api-v2/sipi.md
View file
Expand Up @@ -41,7 +41,7 @@ are described below.
The `upload.lua` script is available at Sipi's `upload` route. It processes one
or more file uploads submitted to Sipi. It converts uploaded images to JPEG 2000
format, and stores them in Sipi's `tmp` directory. The usage of this script is described in
[Creating File Values](../../../03-apis/api-v2/editing-values.md#creating-file-values).
[Upload Files to Sipi](../../../03-apis/api-v2/editing-values.md#upload-files-to-sipi).

Each time `upload.lua` processes a request, it also deletes old temporary files
from `tmp` and (recursively) from any subdirectories. The maximum allowed age of
Expand Down Expand Up @@ -94,7 +94,7 @@ If it encounters an error, it returns `SipiException`.
to create or change a file value. The request includes Sipi's internal filename.
3. During parsing of this JSON-LD request, a `StillImageFileValueContentV2`
is constructed to represent the file value. During the construction of this
object, a `GetImageMetadataRequestV2` is sent to `SipiConnector`, which
object, a `GetFileMetadataRequestV2` is sent to `SipiConnector`, which
uses Sipi's built-in `knora.json` route to get the rest of the file's
metadata.
4. A responder (`ResourcesResponderV2` or `ValuesResponderV2`) validates
Expand Down
46 changes: 7 additions & 39 deletions docs/07-sipi/setup-sipi-for-knora.md
View file
Expand Up @@ -45,11 +45,10 @@ Whenever a file is requested from Sipi (e.g. a browser trying to
dereference an image link served by Knora), a preflight function is
called. This function is defined in `sipi.init-knora.lua` present in the
Sipi root directory. It takes three parameters: `prefix`, `identifier`
(the name of the requested file), and `cookie`. File links created by
Knora use the prefix `knora`, e.g.
`http://localhost:1024/knora/incunabula_0000000002.jp2/full/2613,3505/0/default.jpg`.
(the name of the requested file), and `cookie`. The prefix is the shortcode
of the project that the resource containing the file value belongs to.

Given these information, Sipi asks Knora about the current's users
Given this information, Sipi asks Knora about the current's users
permissions on the given file. The cookie contains the current user's
Knora session id, so Knora can match Sipi's request with a given user
profile and determine the permissions this user has on the file. If the
Expand All @@ -60,8 +59,8 @@ refuses to serve the file. However, all of this behaviour is defined in
the preflight function in Sipi and not controlled by Knora. Knora only
provides the permission code.

See [Sharing the Session ID with Sipi](sipi-and-knora.md#sharing-the-session-id-with-sipi) for more
information about sharing the session id.
See [Authentication of Users with Sipi](sipi-and-knora.md#authentication-of-users-with-sipi) for more
information about sharing the session ID.

## Using Sipi in Test Mode

Expand All @@ -76,42 +75,11 @@ $ docker run --rm -it --add-host webapihost:$DOCKERHOST -v $PWD/config:/sipi/con
```

Then always the same test file will be served which is included in Sipi. In test mode, Sipi will
not aks Knora about the user's permission on the requested file.

## Using Sipi in production behind a proxy

For SIPI to work with Salsah1 (non-angular) GUI, we need to define an additional set of
environment variables if we want to run SIPI behind a proxy:

- `SIPI_EXTERNAL_PROTOCOL=https`
- `SIPI_EXTERNAL_HOSTNAME=iiif.example.org`
- `SIPI_EXTERNAL_PORT=443`

These variables are only used by `make_thumbnail.lua`:

```lua
server.log("make_thumbnail - external_protocol: " .. get_external_protocol(), server.loglevel.LOG_DEBUG)

server.log("make_thumbnail - external_hostname: " .. get_external_hostname(), server.loglevel.LOG_DEBUG)

server.log("make_thumbnail - external_port: " .. get_external_port(), server.loglevel.LOG_DEBUG)

answer = {
nx_thumb = dims.nx,
ny_thumb = dims.ny,
mimetype_thumb = 'image/jpeg',
preview_path = get_external_protocol() .. "://" .. get_external_hostname() .. ":" .. get_external_port() .."/thumbs/" .. thumbName .. "/full/max/0/default.jpg",
filename = tmpName, -- make this a IIIF URL
original_mimetype = submitted_mimetype.mimetype,
original_filename = filename,
file_type = 'IMAGE'
}
```

not ask Knora about the user's permission on the requested file.

## Additional Sipi Environment Variables

Additionaly, these environment variables can be used to further configure sipi:
Additionally, these environment variables can be used to further configure Sipi:

- `SIPI_WEBAPI_HOSTNAME=localhost`: overrides `knora_path` in Sipi's config
- `SIPI_WEBAPI_PORT=3333`: overrides `knora_port` in Sipi's config
Expand Down

0 comments on commit a824bcc

Please sign in to comment.