Meta-discussion: extensibility

[handrews]

I'm filing this issue to take a step back and look at the big-picture questions of extensibility:

• Is extensibility for use cases such as those below in-scope for jschon? Both yes and no are totally reasonable answers here.
• If so, where is the border between jschon and other packages built on it?
• What should people extending jschon expect to be able to do or not do?

The idea of extending JSONPointer and RelativeJSONPointer with subclasses has already been accepted and mostly implemented, but we decided to only do that so far in #103.

Deciding this will help me avoid taking up time with out-of-scope suggestions. And if this is in-scope, help with producing a comprehensive and consistent solution.

Use cases

• Formats such as OpenAPI, AsyncAPI, and the W3C's Web of Things Thing Description embed JSON Schemas in multiple places within the larger document format
• Many "$ref"-handling tools exist, ranging from pure "$ref" replacement to full support of all JSON Schema draft 2020-12 referencing and identification keywords
• JSON Pointers can be further extended in a number of non-standard but useful ways (my original reason for Extensibility of [Relative]JSONPointer classes? #103)

This discussion is primarily motivated by my work building OpenAPI support for oascomply on top of jschon.

The current version in the main repo ended up paralleling a lot of jschon functionality for non-schema documents. My own fork has a work-in-progress version leveraging a lot more of jschon, but with some awkward workarounds to make it viable.

Functional areas

jschon, in my view, covers three cohesive major areas and two less cohesive minor areas:

• Major:
  • a rather comprehensive JSON utility suite (the JSON, JSONPointer, RelativeJSONPointer, and JSONPatch classes)
  • a system for managing resource loading and reference resolution that is almost JSON Schema-independent (Catalog plus Source and its subclasses- I'll refer to this as the Catalog system)
  • JSON Schema support itself (JSONSchema, Metaschema, Vocabulary, Keyword, etc.)
• Minor:
  • URI utlilites (URI)
  • File loading and parsing utilities (jschon.utils)

Side note: the maintainer of jsonschema has made a referencing library covering a similar area as the Catalog system, although I prefer the boundaries drawn by the Catalog system for several reasons.

Extensibility challenges

• Document structure restrictions
  • only child and grandchild instantiation can be controlled
  • all objects at a single level beneath a parent must be the same class
  • parent objects are not fully usable while instantiating child objects (e.g. if parent == root: or if parent: won't work, although if parent is None: will)
  • schema documents are expected to have a schema at the root
  • issues determining what objects are valid reference targets are related to all of this
• The Catalog only works with schemas, not general JSON resources
  • It's quite easy to get it to work with arbitrary JSON subclasses
• There is no channel in the Catalog system to pass back additional information, such as:
  • this resource was loaded from this URL/filesystem path/whatever
  • here's an extra class constructed from the resource that goes with but does not belong inside the JSON subclass (e.g. a file line and column number map)
• If subclassing is the main mechanism, every (sub)class needs to be able to control what subclass of the other classes it will use, accept, and return
  • Already implemented within JSONPointer and RelativeJSONPointer
• URIs in Python are a nightmare
  • Picking a URI class and encapsulating it is the sane thing to do
  • Both jschon and my oascomply package do this
  • Unfortunately, we need to encapsulate different packages (rfc3986 vs rfc3987) due to different requiremetns
  • Getting the translation points between oascomply.resourceid.*, jschon.uri.URI, rdflib.URIRef, just plain str, and maybe one other package I'm forgetting, has been very challenging and a huge source of bugs
  • I'm not sure how to solve this, as it's an artifact of Python not having a usable or even standards-compliant URI / IRI standard library
  • Subclassing is definitely not sufficient. Maybe a Protocol?

Paths forward?

In another month or two, I should have oascomply in good enough shape that it's worth looking at for lessons learned, assuming there's interest in having jschon support this sort of thing in some way. This is particularly relevant for the document structure stuff, which is the most complex (hence a couple of PRs that I've submitted but retracted or moved to draft as I work with it more).

The most difficult is the URI thing. Which feels a bit petty to complain about, particularly as it's more of a Python ecosystem problem than a jschon problem. But I've spent a lot of effort on it recently. This is an area where the question of setting expectations on what is and is not flexible in jschon is pretty relevant.

I think the rest of the above is reasonably straightforward to fix. What I'm doing in my oascomply fork has worked out a lot of it, although sometimes with less-than-elegant workarounds.

Deciding how much extensibility is in-scope would help me figure out what workarounds to try to harden for robustness in oascomply, vs what could be contributed upstream to jschon.

[marksparkza]

This is an important discussion - thanks for opening it. In answer to your first few questions:

Is extensibility for use cases such as those below in-scope for jschon?

Yes, absolutely.

Although the project tagline refers to jschon as a JSON Schema implementation, I generally think of it more as a JSON + schema toolkit, that may grow indefinitely to support any number and variety of JSON + schema use cases. And I think the development of capabilities to support well-established and emerging standards is especially valuable - both in terms of utility to the wider JSON-* community and in terms of increasing the robustness of the library's internal architecture.

If so, where is the border between jschon and other packages built on it?

I'm not sure that there is a definitive answer to this question, but the use case that originally motivated the development of jschon does provide some insight.

This derived package implements a custom (and not yet properly documented) vocabulary, with implementations for a hierarchy of keywords along with custom schema processing.

The rules for keyword validation and schema processing are not defined by any specification explicitly supported by jschon, and so do not belong in the jschon library itself. However, extensive changes were made to jschon to support the development of this package - including the addition of a JSON Patch implementation along with generalizing of some aspects of keyword construction and schema processing.

What should people extending jschon expect to be able to do or not do?

Any generic capability in support of an external use case is fair game for inclusion in jschon, as long as it does not introduce limitations on existing functionality. Specific / custom rules not defined by an explicitly supported specification should be left out.

Having said that, the list of explicitly supported JSON-* specifications may grow - as with the addition of JSON Patch.

[handrews]

@marksparkza thanks for taking the time to read this and reply so quickly! There's lots to think on, here.

In terms of relatively immediate work, here's how I'm thinking of it:

Existing PRs that are relevant, from highest to lowest priority:

• Resolve PR Instantiate correct [R]JP subclasses #105 to finish the [Relative]JSONPointer extensibility work
• Resolve PR Allow adding a default source with a base URI of None #83, which increases catalog flexibility, supports PR 82, and is a prerequisite for PR 85
• Resolve PRs Allow deferring reference resolution to support mutually referencing bundles #82 and/or Additional/alternative support for complex references #85, as that will clarify the full scope of Catalog referencing and loading behavior, and also makes it easier to customize reference resolution
• Resolve PR Allow changing "$schema" in embedded resources #87, as changing "$schema" while evaluating a schema might be generalizable to evaluating more complex mixed-format documents

New PRs — Each of these is quite straightforward, and only the first one is likely to be tediously largs:

• Give more-or-less all of the non-Vocabulary, non-Keyword classes (except URI, which I'll address in another thread) class attributes to allow subclasses to customize usage [EDIT: Draft posted for feedback on direction as PR Requesting feedback on direction: Cooperative subclassing #111]
• Factor JSON child instantiation out into helper method(s) that subclasses can override [EDIT: I forgot this is already PR Fine-grained JSON document structure control #101]
• Add a document_root cached property to JSON - this is needed for OpenAPI as sometimes we need to get to the root regardless of its type, rather than the document_rootschema or resource_rootschema [EDIT: Posted as PR Add JSON.document_root; test JSON and JSONSchema #109]

I'll (eventually) start new discussion threads for each of the more complex topics.

[handrews]

I forgot that PR #101 actually take care of this:

Factor JSON child instantiation out into helper method(s) that subclasses can override

I will check it over and move it out of draft.

[handrews]

Subclass interdependency thoughts

I've spent some time working with the approach in #111 in the oascomply project, and it has led to a few thoughts. The approach mostly works for what I need and could probably be made to fully work by following through in a few places where I didn't quite catch everything. Despite that, there are some warning signs in it:

• The circular import problem that required @classmethod _set_catalog_cls() can show up in other places, and that way of solving it is not great (it depends on instances checking whether the method has been invoked, which works for instances but not if you need to interact with other class methods before any instance has been constructed)
• It feels like it might make more sense to leverage the Catalog as the general system entry point, which configures certain classes to use, rather than this more global-ish arrangement of inheritance and class variables
• Having Metaschema be a subclass of JSONSchema makes sense, but gets tricky when you have a subclass of JSONSchema, and now Metaschema is a sibling. Should there be a Metaschema subclass that is also a subclass of the JSONSchema subclass? Or should metaschema classes and schema classes be treated a bit differently in that they are all JSONSchema subclasses but aren't expected to be subclasses of each other, which requires handling them differently? (this question feels more like a design smell than the right question to answer)

Some intermediate ideas

Given the above, this problem would benefit from stepping back and looking at why different classes need subclassing, and whether they could just be made more flexible instead. I have been playing with refactoring the jschon classes a bit and have come up with the following:

• JSON stays as it is, aside from making resolve_references() public per Allow deferring reference resolution to support mutually referencing bundles #82 and factoring out sub-document instantiation per Fine-grained JSON document structure control #101
• A class CatalogedJSON(JSON) (or a better name) subclass would factor out the catalog. cacheid, and uri handling from JSONSchema so that it can be generalized for not-entirely-schema-like document formats
• A class JSONSchemaContainer(CatalogedJSON) subclass provides additional features needed for document formats that are or contain JSON Schemas. This mostly means having a metaschema and a validate() method. It does not have an evaluate() method as formats like OAS do not have a general notion of evaluation. So far, it's only the metaschema concept that I've found needs to go here
• JSONSchema would now inherit from JSONSchemaContainer, but generally be the same. In my local fork I've factored some of JSONSchema's __init__() into a protected method that, for example, sets up the catalog an URI things that CatalogedJSON needs; it's not clear to me how important it is to preserve JSONSchema not calling the base __init__(), or how the factors that led to that might mean that a deeper inheritance hierarchy might be a concern
• I have a class OASJSON(JSONSchemaContainer) using this system, and it more or less works - it is certainly an improvement over what I was doing before

A more comprehensive re-think

I don't have time right this moment, but I do have thoughts on a more comprehensive design than the above, and will add a reply to this comment thread when I have a moment to write it up.

The short version: Right now I have OASJSON, OASJSONSchema, and OASCatalog subclasses, plus subclasses of Source that aren't really OAS-specific but implement things that I need for OAS.

I think it might be possible to reduce the need for subclassing down to just class OASJSON(JSONSchemaContainer) by making Catalog, Source, and JSONSchema a bit more flexible (in particular, passing back additional information about resource loading to the resource). That would eliminate the need for a lot of subclass coordination.

I still need to think this through, but reducing the need for subclassing in order to make a useful extension might be a more robust solution that enabling subclassing across the board and managing complex interdependencies.

[handrews]

@marksparkza OK, I have spent two weeks re-thinking comprehensively! 😁

I mentioned when closing the PR with the class attributes for related subclasses that I think we need flexibility more than specialization. This means modularizing some of the functionality and generalizing the schema-centric concepts (although this need not look exactly as proposed).

The basic direction I want to go in to generalize things looks like this:

• JSON gets additional document construction methods:

  • get_itemclass() to allow for more complex child instantiation as not all formats are uniform at a given depth like JSON Schema is
  • instantiate_sequence() and instantiate_mapping() more-or-less as shown in PR Fine-grained JSON document structure control #101
  • document_root as shown in PR Add JSON.document_root; test JSON and JSONSchema #109

• JSONFormat(JSON) (better name ideas very much welcome) is the base class for JSON-based formats with additional syntax/semantics that can be validated according to an associated metadocument; the metadocument and its validating rules can change by embedding one format or metadocument setting within a format as a sub-document of another:

  • metadocuments and validation:
    • metadocument (metaschema)
    • metadocument_uri (metaschema_uri)
    • validate()
  • semantic document tree
    • format_parent (parentschema)
    • document_format_root (almost but not quite document_rootschema from PR Add document and resource rootschema properties #78)
    • embedded_format_root (for JSON Schema, would be nearest parent with both "$id" and "$schema", indicating that the document validation rules change)
    • is_embedded_root() (to generalize the check for changing validating_with in PR Allow changing "$schema" in embedded resources #87)

• JSONResourceMixin handles identity, referencing, and Catalog integration. It is a mixin because it is orthogonal to other concerns and it would be perfectly reasonable to, for example, have a direct JSON subclass that is also a resource but not a JSONFormat:
  * uri, base_uri, canonical_uri
  * resource_root (similar to resource_rootschema from PR Add document and resource rootschema properties #78)
  * is_resource_root() (for JSON Schema would be a check for "$id")
  * catalog, cacheid
  * references_resolved, resolve_references (from PR Allow deferring reference resolution to support mutually referencing bundles #82 - deferred reference resolution is needed for this)
  * initialized_resource() (to be called from subclass constructors)

• EvaluableJSON (again, a better name is welcome) is probably just a protocol or pure abstract base class for the evaluate() method and the idea of a Result. This is mostly to make type hinting work when something that can evaluate() is expected which might not be a subclass of JSONSchema, such as the type of JSONFormat.metadocument. Currently, JSON Schemas are the only in-scope things that can evaluate so I don't want to waste effort with a more complex generalization of the idea.

JSONSchema would inherit from JSONFormat and JSONResourceMixin, and implement or inherit from EvaluableJSON. Various OpenAPI document use cases would subclass/implement different combinations.

---

This would expand the current 3-class hierarchy to 6 classes, including one additional class in the main hierarchy, one mixin, and one protocol-or-abstract-mixin. If that seems like a lot, it is the minimum I think is needed for the required flexibility. At one point I had it broken up into 10 classes! Which was like the 5th normal form of a database or something.

The exact way to implement this depends on why JSONSchema.__init__() avoids calling super.__init__(). @marksparkza is it important to preserve this or otherwise avoid additional method calls in the constructor?

In this approach, the Catalog would get add_resource(), get_resource() etc. methods, where get_resource() would take a callable for instantiating the class (there are scenarios with OpenAPI where you need to inspect the contents before determining the correct class to instantiate).

There are some other aspects around retaining some source loading information that I'll post separately. There are use cases for knowing the retrieval URL of something loaded through the catalog, and for attaching additional metadata such as line and column number information that can be used for better error reporting UX. The interface for that would be added to JSONResourceMixin and would require slight changes to the Source interface.

---

Finally, I don't think there's really a need to subclass JSONPointer and RelativeJSONPointer anymore (although using cls() and type(self)() to instantiate is still a good idea). Part of that is due to the changes you made to the exception hierarchy, and part of that is due to me taking the Relative JSON Pointer Spec to IETF 117 last week and it being fairly likely to be adopted by the HTTP API working group. That will hopefully result in the non-spec-defined features I want getting into the spec and not requiring custom extensions. I can work around the issue in the meantime, which would be better than making jschon more complicated.

[handrews]

@marksparkza feel free to ask for any background information, as I have extensive notes on the entire process of coming up with this. I just wanted to keep it relatively concise. I am also happy to do a write-up of requirements in whatever level of detail you want if you want to see those before deciding if the solution above is the right one.

Also, if there are performance concerns for the JSONSchema class and its interactions with the Catalog, I've thought a bit on how to make those code paths faster than the generic ones.

[handrews]

@marksparkza on the plus side, I simplified the above by realizing that I don't need a JSONFormat that is not also a JSONResource (but JSONResource on its own is useful), so JSON <- JSONResource <- JSONFormat <- JSONSchema can just be a straight inheritance chain.

Also on the plus side, I got JSONResource worked out to handle all possible 2019-09+ ways of setting and accessing URIs, including a few that are a bit tricky to sort out with the current JSONSchema (not wrong, and not impossible, just requiring checking what kind of fragment you have or digging into the keywords).

On the minus side, getting all of that to work in a generic way while still bootstrapping with a metaschema and vocabularies... who the heck wrote the spec for this... ah... right... I wrote that spec... 🤦. I have regrets.

Anyway, I continue to appreciate how jschon's design solves these problems, but generalizing it is proving a bit tricky.

[handrews]

EDIT — sub-thread RESOLVED: URI management is outside of the scope of jschon, extensions such as oascomply will work around that for now and jschon is willing to switch to a "better" (in some sense) URI/IRI library if available in the future. No current library is clearly better than the current strategy (wrapping rfc3986).

---

URI hashing

One thing I've realized is that, within a Catalog instance and all JSONSchema (or other) instances managed by it, all URI implementations must have the same hashing behavior. This probably needs to be done using the string form rather than a tuple the way rfc3986 does it (to which jschon.uri.URI defers). This is because libraries are wildly inconsistent about how they handle certain things, particularly missing components.

The correct rules are:

• The path is always at least an empty string, never None
• Other components are empty strings if their separators are present, but None if they are not, e.g. https://example.com has a fragment of None, while https://example.com# has a fragment of an empty string)

While it's not necessary to use the technically correct rules, it is necessary to be consistent, and Python URI libraries are not remotely consistent. For example, rfc3986 parses file:///foo and file:/foo the same (with an authority of None), while rfc3987 correctly parses the former with an authority of "" and the latter with None.

So either everything has to use the same URI implementation (which in my codebase became a nightmare of translating between three different URI libraries depending on what code was talking to what library), or the URI protocol/interface/whatever has to be very clear on the hashing expectations.

[handrews]

On further examination, since URI libraries are not consistent in either direction (parsing or composing), there is probably not a universally consistent hashing strategy. While most of the places rfc3986 is inconsistent are pretty obscure, the file:/// vs file:/ problem is one that impacts me with oascomply. Right now, I am working around it by normalizing file:/// URIs to file:/ as I parse it, although for certain applications that is not ideal.

>>> import rfc3986, rfc3987
>>> from rfc3986.builder import URIBuilder as Builder6
>>> 
>>> show6 = lambda u: (u.scheme, u.authority, (u.userinfo, u.host, u.port), u.path, u.query, u.fragment) 
>>> show7 = lambda u: (u['scheme'], u['authority'], (u['userinfo'], u['host'], u['port']), u['path'], u['query'], u['fragment'])
>>> 
>>> empty_auth = 'file:///foo'
>>> no_auth = 'file:/foo'
>>> 
>>> empty6 = rfc3986.uri_reference(empty_auth)
>>> no6 = rfc3986.uri_reference(no_auth)
>>> 
>>> empty7 = rfc3987.parse(empty_auth, rule='URI')
>>> no7 = rfc3987.parse(no_auth, rule='URI')
>>> 
>>> show6(empty6)
('file', None, (None, None, None), '/foo', None, None)
>>> show6(no6)
('file', None, (None, None, None), '/foo', None, None)
>>> empty6.unsplit()
'file:/foo'
>>> no6.unsplit()
'file:/foo'
>>> Builder6(scheme='file', host='', path='/foo').finalize().unsplit()
'file:/foo'
>>> Builder6(scheme='file', host=None, path='/foo').finalize().unsplit()
'file:/foo'
>>> 
>>> show7(empty7)
('file', '', (None, '', None), '/foo', None, None)
>>> show7(no7)
('file', None, (None, None, None), '/foo', None, None)
>>> rfc3987.compose(**empty7)
'file:///foo'
>>> rfc3987.compose(**no7)
'file:/foo'
>>> 

[marksparkza]

On the URI issue:

URIs in Python are a nightmare

I agree 😭

I'm not sure how to solve this, as it's an artifact of Python not having a usable or even standards-compliant URI / IRI standard library

My feeling is that this is not a problem for jschon to solve. I would quite happily drop jschon.uri.URI if a better 3rd party (or standard) URI / IRI library were available. I think you might be onto something with the protocol idea. If something like that were developed as a standalone package providing a sensible URI / IRI API as a facade to rfc3986, rfc3987, etc, I'd be interested in using that in jschon instead of jschon.uri.URI.

[handrews]

Cool- I have ambitions of either factoring out the URI/IRI support I'm making for oascomply or (more likely) not worrying about making that too comprehensive and doing a separate library on my own after this contract is done in the fall.

I agree that it's out of scope for jschon, so working around it for now should be fine.

[marksparkza]

@handrews, I've given this a lot more thought and have settled on an outlook for jschon that I'm feeling more comfortable with (in terms of my fluctuating availability and interest in the domain) and that I believe is aligned with the spirit of open source software development. I've updated the project description and contributing guidelines accordingly.

With respect to the more complex modifications that you've proposed in support of the OASComply project, it's unlikely that these will be merged any time soon, if at all. I have some ideas that might substantially alter the jschon internals and API, but which might only begin taking shape in conjunction with further development of use cases in my work environment. My suggestion would be to rename your fork of jschon to any name of your choosing (there's no need to include jschon in the name), and to publish and manage it as you see fit. This may lead to a divergence in the structure and mechanics of our respective forks, but I believe that diversity in implementation is a good thing and may ultimately promote wider adoption of the relevant specifications.

I do apologise for any uncertainty and inconvenience I've caused with my evolving perspective on the direction of jschon. My sincere thanks for your valuable insights and contributions to the project.

[handrews]

@marksparkza this sounds more than reasonable - thank you so much for your flexibility! I do hope we can keep contributions going back and forth when feasible, as I'd love to continue to benefit from your perspective. It might be a bit before I take action on any of this as my OASComply coding is somewhat on hold while we sort out some specification ambiguities regarding referencing. But I expect to either get back to coding next month, or in Q1 of 2024 depending on what the OpenAPI folks want me to deliver next.