How does Introspected REST reduce complexity?

[Flogex]

It’s inflexible, difficult to implement, difficult to test, with performance and implementation issues. But most importantly, any implementation of REST model is very complex.

Now to the reader, it should be obvious that even if we manage to offload some of the aforementioned information to the Media Type, we would still have a very complex, massive, response from the server that mostly includes HATEOAS and not actual data. In our experience, such responses are very hard to implement correctly, test, be performant and even debug. After all, a human will sit down and write the initial code and debugging the code by the eye is important. Simplicity is crucial.

Then, we will propose a new model, Introspected REST, that solves the issues that REST creates and allows the design of progressively evolvable APIs, in a much simpler way than conventional REST.

I have problems understanding how Introspected REST can achieve more simplicity and avoid the issues of REST quoted above. As far as I understand, Introspected REST is built upon two pillars:

1. The client receives hypermedia "on the side", i.e., hypermedia is not included in the actual response, and
2. media types are replaced by composable micro types.

If a client is not interested in hypermedia, Introspected REST makes it easier for this client because it does not have to fetch and parse this unnecessary data. But I can't see how Introspected REST will reduce the complexity – of the client implementation or the response message – for a hypermedia-ready client.

Can you please explain

• what you understand by complexity and
• how Introspected REST can reduce this kind of complexity compared to REST?

[vasilakisfil]

I am not sure if you have read the publication, or just skimmed over it, REST issues according to author are summarized here: https://introspected.rest/#82-deriving-the-need-for-a-new-model and basically these issues what Introspected REST tries to solve.

At the full extend, then yes, REST and Introspected REST have the same complexity, but REST is just a case/instance of Introspected REST and not all clients are interested in fully fledged, hypermedia-driven, version of the API response, reality has shown quite the opposite actually.

Then you might come and say, "but hey that's why we have media types, you can ask a different format" and I would agree with you, the problem is that

1. Roy in his thesis (and subsequent talks/blogs) doesn't embrace the idea of non-hypermedia API formats, according to his opinion, an API client is just another browser (where you click links etc), of course, experience and practice has shown a different reality
2. having a completely separate Media Type for each representation of the API is quite tedious work. In general, creating a new Media Type is a very challenging thing to do (just look at JSON:API) and should be avoided, unless the things you bring in worth the effort (quite unlikely). So having a distinct Media Type for each API representation, is a no go. That's where MicroTypes come in and fill the gap. If you are gonna spend time and money creating a Media Type, at least you should do it in an a) extensible b) composable c) configurable (through introspection) way. The idea is that you build a very solid (fixed) core of your Media Type, and start adding features in small, isolated, composable, configurable (introspectable), negotiable modules. A similar notion (but not of that extent) of MicroTypes is actually used by JSON:API (called extensions).

Such design (Introspected REST though MicroTypes) has many benefits (resembles Software Engineering learning outcomes of decades) and in the long run, each small composable component of the larger Media Type, could be reusable across many different Media Types (just look at pagination, links etc, each Media Type defines those, but in a completely different way).

Finally, I would like to note that such architectural designs like REST, GraphQL, SOAP, Introspected REST, etc should mostly be evaluated when thinking APIs for clients that we don't control, and clients that were built once, and should last decades, even when the APIs evolve. Such architectural designs helps us to deliver reliable remote software, and evolve without breaking things.

If you can control the client (and have the money/time/ownership to update it), then you can even use a cat to deliver the message, no need to fiddle with such architectural design. In reality, even if you control the client, its still probably a good idea to follow a standardized architectural pattern, since it will save you time in the long run. Which one from the ones I mentioned, is up to you and as an architect you need to balance the benefits/drawbacks of each one for your specific use case before you pick one. However, when it comes to clients that you don't control (which could even be an ios app), then you have to choose a sane architectural pattern (and no, a cat isn't future proof), if you don't want to accumulate technical debt. Believe it or not, most software engineers/devs I have met fail to understand the need of evolvability in APIs (maybe because they built APIs that they always (think that they) control? or APIs that they never (have to) maintain? no idea).

Did I answer your question ?

[Flogex]

Thank you very much for your detailed answer.

To summarize my current understanding: Introspected REST is less complex for clients not interested in hypermedia because the responses contain only the actual data and they can just skip the introspection part.

But it also reduces complexity because client and server can negotiate MicroTypes, therefore no two media types are needed to support different kinds of clients, e.g., clients that want hypermedia vs. client that don't. Clients and servers don't have to implement a completely new media type from scratch, but can ideally reuse and compose existing libraries for specific MicroTypes.

Have I got this right?

[vasilakisfil]

On your first point yeah.

On your second point still yes, but you scratch the surface. MicroTypes give you many advantages, apart the one you mentioned (which is the most distant benefit, meaning, it's a benefit when everything else is in place). But in order to understand the advantages, you need to first understand how Media Types work.

Media Types, according to our beloved RFCs at least, are the manuals for the clients. If a client implements a Media Type, it means that it understand a response with content-type that media type. The problem is that, nowadays, (json based) APIs don't serve only data, but they provide tons of other affordances: things like, hypermedia (links, actions, forms), they could support JSON-LD or in general some RDF stuff, pagination (which is not only the linking part, but also the client asking part), embedded resources (also known compound documents a la JSON:API), transformations etc..

So a Media Type designer has an enormous task when designing a new Media Type. These stuff are not easy to specify in a single run. Also it's not easy to communicate the information/affordances that the Media Type provides. And of course its not easy to implement all at once on the client-side. In REST, when having such feature-complete Media Type, you would either have to a) serve everything in the same response b) serve the data in the main response with links of each metadata (MicroType equivalent) in the response. For (a) the benefits of Introspected REST are obvious (it's the first point you mentioned). For (b) the benefits of Introspected REST are not so obvious.

The first clear benefit is on the Media Type designer, and to some extend to the client who follows that Media Type: In REST you have to define all metadata (MicroTypes equivalent) types (like, hypermedia, JSON schemas, JSON-LD etc) in the same Media Type, all at once. As I said, that's a lot of tedious work. By having a tight core and small modules that compose the overall Media Type, things are a bit easier since each module will follow the Single Responsibility Principle, that we know it works well as a pattern. Also, by using introspection on Media Types that need to serve metadata, you can serve the configuration (=metadata) of each MicroType without bloating the response, for those clients who really want it and really need it. But the hidden benefit of MicroTypes is evolvability: a conventional Media Type cannot be evolved. Let's say that you have created/specified a Media Type called foo, but then let's say that you want to add support for https://jsonforms.io/ somehow. That's impossible. You cannot re-open the Media Type once it's published/defined. So by defining a Media Type that is evolvable by default, through MicroTypes, makes the Media Type to always be in-fashion, even when patterns and designs change/evolve.

This architectural design unlocks huge potentials on building automated clients from scratch that are reliable without hindering your API design. While in REST, you would have to re-write the API in a new Media Type in cases that your core data are the same, but you wanted to support small improvements, like better pagination, in Introspected REST, you just need to deploy/employ a new MicroType with your base Media Type and let clients that support it, ask for it.

[Flogex]

Thank you! This gave me new insights.