i18n and n11n of resource identifiers

[woutermont]

This PR updates the Solid protocol with a normative section on the use of Internationalized Resource Identifiers (IRIs).

It attempts to address the following issues (and possibly other related ones):

• Use IRIs for resource identification #347
• Add URI Normalization #22
• Clarify whether ACL needs normalization web-access-control-spec#93

The gist of it is as follows:

• Require Solid servers to only mint IRIs for their resources according to a normalized form.
• When writing HTTP messages metadata, transform IRI to URI.
• When reading HTTP messages metadata, transform URI to IRI and normalize.
• Always leave the data itself (content) alone.

What I did not (yet) do:

• Change mentions of URI to IRI throughout existing text.
• Adhere to W3C i18n best practices: WHATWG URL and UTF-8 as additional restrictions.

Changes

Commit	Target	Correction class	Note
65df99b	#uri	2	rename to #identifiers ("Resource Identifiers")
65df99b	#identifiers	2	rename from #uri ("Uniform Resource Identifier")
65df99b	#toc	1	toc update
98cad8b	#bib-rfc3987	1	add rfc3987 to normative references
5974359	#bib-rfc3987	1	add rfc3987 to normative references
e679fe0	#iri	2	add iri definition to terminology
1e5b0bd	#bib-uax15	1	add uax15 to normative references
dd2c455	#bib-ascii	1	add ISO/IEC 646:1991 to informative references
b12fa2e	#uniform-resource-identifier	2	update uri definition in terminology
63c7917	#uniform-resource-identifier	1	correct bib ref
6a225a0	#iris	4 (new requirement)	add section requiring IRIs
6a225a0	#toc	1	toc update

HMLT Preview | HTML Diff

[csarven]

What do you think about reusing the language from 3987 (section 3.2's "Converting"):

Suggested change

<p><span about="" id="server-iris-from-http" rel="spec:requirement" resource="#server-iris-from-http"><span property="spec:statement">When interpreting a URI in an <a href=#http>HTTP</a> message, except in the content, as a resource identifier, a Solid <span rel="spec:requirementSubject" resource="#Server">server</span> <span rel="spec:requirementLevel" resource="spec:MUST">MUST</span> map the URI to an IRI according to the algorithm provided by [<cite><a class="bibref" href="#bib-rfc3987">RFC3987</a></cite>] (<a href="https://datatracker.ietf.org/doc/html/rfc3987#section-3.2">section 3.2</a>), and normalize the resulting IRI to the <a href="#iris-norm">normal form</a> provided in this section.</span></span></p>

<p><span about="" id="server-iris-from-http" rel="spec:requirement" resource="#server-iris-from-http"><span property="spec:statement">When interpreting a URI in an <a href=#http>HTTP</a> message, except in the content, as a resource identifier, a Solid <span rel="spec:requirementSubject" resource="#Server">server</span> <span rel="spec:requirementLevel" resource="spec:MUST">MUST</span> convert the URI to an IRI according to the algorithm provided by [<cite><a class="bibref" href="#bib-rfc3987">RFC3987</a></cite>] (<a href="https://datatracker.ietf.org/doc/html/rfc3987#section-3.2">section 3.2</a>), and normalize the resulting IRI to the <a href="#iris-norm">normal form</a> provided in this section.</span></span></p>

Or do you think that re-using the wording "map" from #server-iris-to-http would be simpler and used more liberally.

[csarven]

What do you think about leaving out the section #iris-norm? Would conforming to 3987's sections 3.1-3.2 (#server-iris-to-http, #server-iris-from-http ) suffice?

Suggested change

	<ul id="iris-norm">
	<li id="iris-nfc">The IRI is a Unicode string in Normalization Form C (NFC) [<cite><a class="bibref" href="#bib-uax15">UAX15</a></cite>].</li>
	<li id="#iris-unreserved">The IRI does not contain percent-encoding triplets corresponding to <em>unreserved</em> characters.</li>
	<li id="iris-hex">Hexadecimal digits within percent-encoding triplets corresponding to <em>reserved</em> characters are represented using <em>uppercase</em> letters.</li>
	<li id="iris-scheme">The <code>http</code> or <code>https</code> scheme of the IRI is represented using <em>lowercase</em> characters.</li>
	<li id="#iris-host">The host of the IRI is represented using <em>lowercase</em> characters.</li>
	<li id="iris-port">If the port of the IRI is the default port for its scheme, the port subcomponent is left out.</li>
	<li id="iris-path">The path of the IRI does not contain dot-segments.</li>
	</ul>

[woutermont]

Not a good idea. I specifically wrote this out explicitly because [A] the conversion in RFC3987 is not deterministic (it leaves certain choices, which is a one of the big issues opposers like WHATWG have with it), and [B] these steps should be compatible with both RFC3987 and WHATWG URL. Of course, in practices, a single deterministic URI/IRI spec would exists to which we can simply refer, but it doesn't.

[csarven]

When a server conforms to 3987's sections 3.1-3.2 (#server-iris-to-http , #server-iris-from-http), does it satisfy the #server-iris-norm requirement? If so, can #server-iris-norm be removed?

Suggested change

	<p><span about="" id="server-iris-norm" rel="spec:requirement" resource="#server-iris-norm"><span property="spec:statement">A Solid <span rel="spec:requirementSubject" resource="#Server">server</span> <span rel="spec:requirementLevel" resource="spec:MUSTNOT">MUST NOT</span> create IRIs that do not conform to this normal form.</span></span></p>

[woutermont]

As clarified in reply to your other comment, it does not in all cases.

[csarven]

Thank you, good stuff! I think this PR captures the essence and needs of the referenced issues. As I understand it, this PR (or in this direction) would satisfy https://www.w3.org/TR/international-specs/#resid_what_to_spec .

It would be good to document some implementation experience. What currently conforms to this? How complex is it implement?

Is there a reason to introduce additional restrictions to WHATWG URL and UTF-8 ( https://www.w3.org/TR/international-specs/#resid_what_to_spec_protocol mentions )?

[woutermont]

As I understand it, this PR (or in this direction) would satisfy https://www.w3.org/TR/international-specs/#resid_what_to_spec.

Yes and no. It is compliant with the therein prefered specs, but our text does not mention WHATWG URL, which the iSpecs document demands. I have not done so because it really does not help clarity at all.

[damooo]

It attempts to address the following issues (and possibly other related ones):

• Use IRIs for resource identification #347
• Add URI Normalization #22
• Clarify whether ACL needs normalization web-access-control-spec#93

There are two orthogonal issues here. One is about normalization of identifiers, and other is about allowing iris as identifiers for solid resources.

Though it is important to address identifier-normalization, it is not going to be correct/unambiguous/practical to allow iris as identifiers for solid resources. There are many issues. I will specify a few.

1. Incompatible with http & web tools

Solid protocol is a profile of http. Http allows only uris for resources. Thus, as all Solid resources are http resources, their identifiers must also be uris. Otherwise Solid will not be compatible with much of the web and most of the tools.

For example any generic link traverser (like rdflib/tpf's/communica/...) don't kmow these random mappings, and directly use uri as the identifier for resource they fetched.

Many tools like rdf parsers, jsonld processors use http resource uri as the base uri for processing representations. Thus it is going to be nightmare to deal with all existing generic tools.

Basically, if we cross the semantics of http, solid is no longer going to be a profile of it, and none of the http tools work seamlessly.

2. Impossible to specify

Essentially, it is not possible to specify about these mappings with sufficient rigour. I will specify few fundamental issues in proposed spec for example.

Proposed:

When interpreting a URI in an HTTP message, except in the content, as a resource identifier, a Solid server MUST map the URI to an IRI according to the algorithm provided by [RFC3987] (section 3.2), and normalize the resulting IRI to the normal form provided in this section.

First of all, resource servers doesn't even recieve a URI for target resource in a http request message. And what server receives as part of message is also varies between http versions. A request essentially contains only request-target in a message in one of allowed forms (asterik-/origin-/absolute-/authority- forms). It is server's work to reconstruct final resource uri using request-target hint and it's own custom routing config, as per its discretion. As rfc 9110 says:

Upon receipt of a client's request, a server reconstructs the target URI from the received components in accordance with their local configuration and incoming connection context. This reconstruction is specific to each major protocol version. For example, Section 3.3 of [HTTP/1.1] defines how a server determines the target URI of an HTTP/1.1 request.

Thus resolved resource uri will never be part of the message, and thus proposed statement has no effect.

Along with that, there can be relative uri-references, which are not uris.

And the most serious issue is, given a uri in a message (e.g in Link headers), we don't know if it refers to a solid resource or not. Thus we can never be sure whether to apply such a transform.

These issues are just on the surface, and already breaks a lot. Thus it may be good, if we address only uri-normalization restriction part only.

@woutermont, @csarven

[jeff-zucker]

A somewhat related issue. The WebID 1.0 spec says this :

A WebID is a URI with an HTTP or HTTPS scheme ...".

Would that need to be modified to accomodate IRIs?

[woutermont]

Thanks @damooo, for the elaborate comment, which raises some good questions. I will try to address them all below.

[@damooo:] There are two orthogonal issues here. One is about normalization of identifiers, and other is about allowing iris as identifiers for solid resources.

True, there are two issues here; but they are not orthogonal. The choice for URI, IRI and/or WHATWG URL as identifiers is tightly bound with the normalization procedures, since it is the specifications of these identifiers that specify the possible procedures.

---

Compatibility with HTTP

[@damooo:] Solid protocol is a profile of http. Http allows only uris for resources. Thus, as all Solid resources are http resources, their identifiers must also be uris. Otherwise Solid will not be compatible with much of the web and most of the tools.

To be entirely correct, there is no such thing as an "HTTP resource" (apart from the trivial fact that HTTP URIs are URIs, which refer to resources).

But to address your concerns about compatibility: any resource can be identified by more than one identifier. This is how the IRI specification maintains compatibility with URI: it provides a unique mapping from IRIs to URIs. Every HTTP IRI therefore has an HTTP URI that refers to the same resource.

[@damooo:] For example any generic link traverser (like rdflib/tpf's/communica/...) don't kmow these random mappings, and directly use uri as the identifier for resource they fetched. [...] Many tools like rdf parsers, jsonld processors use http resource uri as the base uri for processing representations. Thus it is going to be nightmare to deal with all existing generic tools.

Given the existance of the mapping, as explained above, this danger is avoided: HTTP programs can simply continue to use the corresponding (mapped) URI. They do not need to know anything about IRI or the IRI<->URI mapping. It is the IRI-capable server that does the mapping upon entry and exit of messages.

[@damooo:] Basically, if we cross the semantics of http, solid is no longer going to be a profile of it, and none of the http tools work seamlessly.

As concluded above, this is not true. Moreover, I would contend the idea that "Solid is a profile of HTTP." A specification profile is an extension point defined by the specification itself (see the W3C Note on Variability in Specifications). HTTP, however, does not define such an extensoin point; "HTTP profiles" do not exist. Solid is a separate, standalone specification that uses HTTP as a communication protocol. Given the compatibility of HTTP and IRI, as sketched above, it is thus perfectly possible to combine that with using IRI as an identification scheme.

---

URI<->IRI mapping

[@woutermont (PR proposal):] When interpreting a URI in an HTTP message, except in the content, as a resource identifier, a Solid server MUST map the URI to an IRI according to the algorithm provided by [RFC3987] (section 3.2), and normalize the resulting IRI to the normal form provided in this section.

[@damooo:] First of all, resource servers doesn't even recieve a URI for target resource in a http request message. [...] A request essentially contains only request-target in a message in one of allowed forms (asterik-/origin-/absolute-/authority- forms). It is server's work to reconstruct final resource uri using request-target hint and it's own custom routing config, as per its discretion. [...] Thus resolved resource uri will never be part of the message, and thus proposed statement has no effect.

What you describe is correct: an HTTP server reconstructs the URI of the target resource based on some algorithm (that we can treat as a blackbox here). However, this does not contradict my proposal (although it points out that it should probably be rephrased): when I say "a URI in an HTTP message", I do mean "the URI of the target resource, as reconstructed by the server." According to my proposal, given that URI, a Solid server should then immediately map it to its corresponding IRI, thereby forming the boundary between the URI-based communication channel (i.e. HTTP) and the IRI-based resource storage (i.e. Solid).

[@damooo:] Along with that, there can be relative uri-references, which are not uris.

The 'blackbox' algorithm the server applies to identify the target resource will always result in an URI; i.e. the algorithm includes the logic for going from a URI reference to a URI (typically by prepending a base URI). This is described in RFC 9110 § 7.1 Determining the target resource (¶ 2).

[@damooo:] [G]iven a uri in a message (e.g in Link headers), we don't know if it refers to a solid resource or not. Thus we can never be sure whether to apply such a transform.

This is true, but the proposal only applies to the target URI. Any other URI in the request (Link headers, redirects etc.) is left alone, since they will typically be used in subsequent HTTP requests. It is not the job of Solid to interfere in that.

---

Compatibility with WebID

I will also shortly address @jeff-zucker's question:

[@jeff-zucker:] The WebID 1.0 spec says this :

A WebID is a URI with an HTTP or HTTPS scheme ...".

Would that need to be modified to accomodate IRIs?

No, that is not necessary. Since any URI is also an IRI, WebIDs can still be Solid resources. For some time, WebIDs will then not benefit of the extension of Solid to IRI, but I suspect that eventually WebID will also make the move, so that WebIDs can also include international characters. In any case, using IRIs in Solid will not break existing usage of WebIDs.

---

Hope to have addressed your issues/questions adequately. They shed some missing clarity that I will try to add to the phrasing of the proposal. Thanks for that.

[jeff-zucker]

Would that [WebID 1.0] need to be modified to accomodate IRIs?

No, that is not necessary. Since any URI is also an IRI, WebIDs can still be Solid resources. For some time, WebIDs will then not benefit of the extension of Solid to IRI, but I suspect that eventually WebID will also make the move, so that WebIDs can also include international characters. In any case, using IRIs in Solid will not break existing usage of WebIDs.

But since the Solid WebID Profile spec does not define a WebID, rather counts on WebID 1.0 to define it, we would not be able to remain spec compliant if we use IRIs in WebIDs.

[woutermont]

But since the Solid WebID Profile spec does not define a WebID, rather counts on WebID 1.0 to define it, we would not be able to remain spec compliant if we use IRIs in WebIDs.

No. We we would only break compliance if we use non-URI IRIs as a WebID. So we simply continue to mint only URIs for WebIDs, until WebID also supports IRIs. This is not an issue.

[damooo]

Given the existance of the mapping, as explained above, this danger is avoided: HTTP programs can simply continue to use the corresponding (mapped) URI. They do not need to know anything about IRI or the IRI<->URI mapping. It is the IRI-capable server that does the mapping upon entry and exit of messages.

That is an issue. Thus, all the http tools determine the uri as the identifier.

And they will use that as the base uri, to resolve the resource representations. Thus any relative identifiers in a rdf resource will be resolved against the uri by these clients. While server uses the iri for the resolution. This will soon lead to chaos, as clients also start writing. For a single resource, some places an iri will be used, some places a uri. Some places absolute, some places relative. We cannot deterministically do anything with that data. This is what i meant to be incompatible.

This is true, but the proposal only applies to the target URI. Any other URI in the request (Link headers, redirects etc.) is left alone, since they will typically be used in subsequent HTTP requests. It is not the job of Solid to interfere in that.

That is an issue. Solid uses link headers to link to other resources (acls, descriptions, storage root, ....). If their identifiers are iris, there is no way to send them in the links, as link headers accept only uris. If we encode as uris and send, then client will deduce the uri as identifier for acls, etc. and uses them when writing statements. While server using iris. Repeat of the above problem. With severe security consequences.

[woutermont]

@damooo, the concerns you raise are interesting questions, but should not pose an issue for the adoption of IRIs.

[T]he proposal only applies to the target URI. Any other URI in the request (Link headers, redirects etc.) is left alone, since they will typically be used in subsequent HTTP requests. It is not the job of Solid to interfere in that.

That is an issue. Solid uses link headers to link to other resources (acls, descriptions, storage root, ....). If their identifiers are iris, there is no way to send them in the links, as link headers accept only uris. If we encode as uris and send, then client will deduce the uri as identifier for acls, etc. and uses them when writing statements. While server using iris. [...]

This is not a problem. I should correct my own clarification though: what I meant with "left alone" was that the Solid spec should not do unnecessary mappings. However, I seem to have implied that thus no mappings should take place except for the target URI. That is not the case.

Link headers, to address your case, are per definition mapped IRIs, so the server should map IRIs to URIs when adding such headers, and any client following Web Linking (RFC8288) correctly should map that URI back to an IRI again before use.

[...] HTTP programs can simply continue to use the corresponding (mapped) URI. They do not need to know anything about IRI or the IRI<->URI mapping. It is the IRI-capable server that does the mapping upon entry and exit of messages.

[If] all the http tools determine the uri as the identifier [then] they will use that as the base uri, to resolve the resource representations. Thus any relative identifiers in a rdf resource will be resolved against the uri by these clients. While server uses the iri for the resolution. This will soon lead to chaos, as clients also start writing. For a single resource, some places an iri will be used, some places a uri. Some places absolute, some places relative. We cannot deterministically do anything with that data. This is what i meant to be incompatible.

Resolution is a more complex concern, since i.m.o. IRI resolution in the RDF spec is underdefined. However, the way you frame it misses the point. When I say "HTTP program" I mean the code that performs HTTP requests. Any client that can follows the RDF spec to parse a document knows that RDF contains IRIs rather than URIs, and thus that for resolution and dereference it should perform the necessary mappings. In practice, this is easy, since almost all those clients rely on HTTP libraries following the WHATWG Fetch spec, which can handle requests for any URL (whether it is a URI or an IRI). For any direct request, this suffices: the Solid app knows the base IRI for resolution within RDF documents, and can map to URI for dereferencing.

However, I am very glad you brought up this concern around resolution, because there seems to be something missing with regard to redirected requests: when an app is redirected from somewhere, it does not know whether the resource it ends up with originates from a Solid server or not (at least not without additional checks), and thus does not know whether to interpret the Location header as a URI or an IRI. There seem to be a number of more and less elegant solutions to this:

1. Let the client detect and follow the link from the response to the resource's storage and its storage description. This is rather convoluted, but possible with the current spec. It does require clients to know this thouh, so non-Solid clients accessing public RDF resources on a Solid server will resolve relative IRI references wrongly.
2. Let the server declare in a response header that it is a Solid resource, e.g. with a Link header, as mandatory X-Powered-By, or with some other header. This is more elegant but adds an extra requirement to the spec, and still provides no support for non-Solid clients.
3. Let the server add a base directive to RDF representations that support relative references. This is easy to implement and has the added benefit of working out-of-the-box for non-Solid clients as well.

My preference goes out to the last option. What do you think, @damooo?

---

An example

The following example describes a scenario using IRIs in Solid, using the base directive as a solution to the resolution issue. To make it somewhat realistic, I assume WebIDs can also be IRIs (which is still an ongoing effort); but one could rewrite the example using any other RDF resource instead.

The example:

I use SolidRock, a music app built on Solid, and want to share a song with my Ukranian friend Ivan. He is not yet in my contacts, so I copy his WebID https://id.ua/Іван, which is an IRI, and paste it in the app. SolidRock tries to fetch the WebID document. The HTTP library used by the app follows the WHATWG Fetch spec and maps Ivan's WebID to the URI https://id.ru/%D0%86%D0%B2%D0%B0%D0%BD, before performing a GET request.

IDUA (https://id.ua/) is the Ukranian WebID minter Ivan registered his WebID with. At some point in the past, Ivan configured the service to store his WebID Document on his Solid storage pod https://сховищу.ua/Іван/id. In order to respond to the request of the SolidRock app, IDUA maps the WebID Document IRI to the URI https://xn--b1amqlni5a.ua/%D0%86%D0%B2%D0%B0%D0%BD/id, and includes it in the Location header of a 303 response. Upon receiving this 303 response, the fetch library automatically makes another HTTP GET request to the given Location URI.

Сховище.ua, Ivan's Solid storage provider, then receives the request and maps it back to the corresponding IRI https://сховищу.ua/Іван/id (following my proposal in this PR). Since this is a public resource, the server sends back a 200 response including the following Turtle representation of the WebID Document, adding the resource's identifier as base IRI (following my proposal earlier in this comment):

@base <https://сховищу.ua/Іван/id> .
@prefix Іван:    <https://id.ua/Іван> .
@prefix foaf:    <http://xmlns.com/foaf/0.1/> .
@prefix solid:   <http://www.w3.org/ns/solid/terms#> .
@prefix ldp:     <http://www.w3.org/ns/ldp#> .

<> a foaf:PersonalProfileDocument ;
    foaf:primaryTopic Іван: .

Іван: a foaf:Person ;
    foaf:name "Ivan Petrovich Sidorov" ;
    solid:oidcIssuer <https://login.id.ua/> ;
    ldp:inbox <./вхідні> .

Furthermore, the Solid server also includes a Link header pointing to the storage pod. While Link headers are IRIs, they must be converted to URIs for transfer over HTTP:

Link: <https://xn--b1amqlni5a.ua/%D0%86%D0%B2%D0%B0%D0%BD/>; rel="http://www.w3.org/ns/pim/space#storage"

Before sending Ivan a notification, the SolidRock app needs to check some capabilities of his Solid server. It extracts the URI from the Link header, and directs the fetch library to perform a HEAD request (skipping the redundant mapping to and from IRI here for performance). The Solid server receives the request and in computing the target resource it maps the URI to the normalized IRI https://сховищу.ua/Іван/. It sends back a 204 response with a Link header of rel=solid:storageDescription, and the same process is performed again to retrieve that Storage Description.

After learning whatever it needed to know about the server's capabilities, the SolidRock app parses Ivan's WebID document using any Turtle parser. It then queries the parsed RDF data for an ldp:inbox triple linking Ivan's WebID to an inbox. Resolving the resulting relative IRI reference using the base directive, the RDF API of the parsed Turtle returns inbox IRIhttps://сховищу.ua/Іван/вхідні.

Having found the inbox IRI, the SolidRock app directs its fetch library to POST a notification to it, containing whatever data SolidRock uses to share songs. Before performing this HTTP request, the fetch library maps the inbox IRI to a URI.

[elf-pavlik]

The proposed text looks equivalent to https://www.w3.org/TR/rdf12-concepts/#section-IRIs

Since Solid heavily builds on RDF, we should closely follow the latest specifications.