Validation of endpoints against a given OpenAPI schema

[adamw]

Sometimes it might be desirable to verify if the endpoints match a given OpenAPI schema. To implement this, we would need to parse the schema (see softwaremill/sttp-apispec#2 - although this is already partially done), and then invoke comparison of the two schemas (generated from endpoints & parsed), ignoring fields which do not matter (e.g. descriptions, or examples)

[ghik]

What's the "user story" here? Is it something like:

1. User has a given OpenAPI to implement
2. Decides to do it manually instead of using codegen
3. Writes an API in tapir but wants to validate it against the given OpenAPI

?

[adamw]

This, or: develops an API, publishes it, and wants to ensure that anything that's changed doesn't break the published contract

[ghik]

softwaremill/sttp-apispec#157 introduced SchemaComparator to check compatibility between schemas. This is likely the most complex part of full OpenAPI validation. Schema comparator would be used:

• to check compatibility of inputs (parameters, request bodies) - schemas from previous API version (i.e. client's) would be used as "writer" schemas, while schemas from current API version (i.e. server's) would be used as "reader" schemas
• to check compatibility of outputs (responses) - schemas from current API version (i.e. server's) would be used as "writer" schemas, while schemas from previous API version (i.e. client's) would be used as "reader" schemas

Possible improvements to SchemaComparator:

• Make it understand more Schema patterns and JSON Schema keywords (see SchemaComparator sttp-apispec#157 for more details on comparator's assumptions and limitations)
• Make it customizable with pluggable validation logic
  The comparator may be too conservative in some cases. For example, it might not understand that some format values are compatible, or that some particular schema patterns are compatible in specific use cases. It may be worth to open SchemaComparator's implementation to extensions which would allow the user to override parts of its logic, in order to fine-tune it to their needs and avoid false negatives.
• Possibly rethink the structure of compatibility issues being reported (e.g. by making it completely flat or completely tree-like, deduplicating issues for heavily reused named schemas etc.). Optimal structure largely depends on how exactly we want these results to be presented to the user.

Further work needed to have a full OpenApi comparator:

• Make sure OpenAPI can be deserialized (it looks to me like all the decoders are already there, but I haven't tested them)
• Implement an OpenAPIComparator similar to SchemaComparator.
  This requires comparing the following OpenAPI objects: MediaType, Encoding, RequestBody, Responses, Response, Header, Parameter, Operation, PathItem, and ultimately OpenAPI itself.

[ghik]

Proposed structure for compatibility issues reported by full OpenAPI comparator

• missing path - server API does not define a path item that client API expects
• incompatible path - there are incompatibilities in operations for given path item
  • missing operation for given HTTP method
  • incompatible operation for given HTTP method
    • a required server operation parameter is missing or optional in client operation
    • client parameter is incompatible with corresponding server parameter
      • client parameter schema is incompatible with server parameter schema
      • client content is incompatible with server content (see below for details)
      • style or explode don't match between client and server
      • allowEmptyValue is true for client but false for server
      • allowReserved is true for client but false for server
    • request body is required in server operation but is missing or optional in client operation
    • client request body is incompatible with server request body
      • client content is incompatible with server content (see below for details)
    • server responses are incompatible with client responses
      • response for some status code or range is defined by the server but not by the client (note: response for a particular code may be defined using a range or default)
      • response for some status code or range defined by the server is not compatible with corresponding definition on the client
        • server response content is incompatible with client response content (see below for details)
        • a response header is required by the client but is missing or optional on the server
        • server response header is incompatible with client response header
          • server header schema is incompatible with client header schema
          • server header content is incompatible with client header content (see below for details)
          • style or explode don't match between server and client
          • allowEmptyValue is true for server but false for client
          • allowReserved is true for server but false for client
    • TODO: callbacks, security?

Compatibility issues for content comparison:

• media type is defined in writer's content but not defined in reader's content
• writer media type is incompatible with reader media type
  • writer schema does not match reader schema
  • TODO: encoding?

Note: "writer" and "reader" correspond to:

• client and server, when comparing input (request) definitions
• server and client, when comparing output (response) definitions