Missing support to describe API for services like Google Endpoints

[mleclercq]

Google endpoints is an API management service. It allows to make REST APIs publicly accessible while taking care of some security features like API Key validation, JWT authentication, quota enforcements etc. It works using a reverse proxy in front of a backend server. These features are configured by an OpenAPI document.

I tried to see if I could use endpoints4s to describe a REST API and get the appropriate OpenAPI document to be used to configure google endpoints. I faced a couple of challenges, most of them related to the current OpenAPI support:

• Google endpoints still doesn't support OpenAPI 3.0...
• Endpoints4s doesn't support OpenAPI extensions while several features of google endpoints are configured using such extensions (OpenAPI extensions are not supported #671).
• The support SecurityRequirement and SecurityScheme in endpoints4s OpenAPI model is limited while these are needed to describe API Key validations and JWT authentication.

Also different endpoints in a given REST API can use different set of google-endpoints features, I needed a way to express these features individually and the ability to compose them (#794).

So I tried to tackle these different issues and I managed to get something that seems to be working:

• I fixed the support for SecurityRequirement and SecurityScheme
  mleclercq/endpoints@master...security-schemes
  This also completes the support for the Info object according the the OpenAPI 3.0 spec.
• I added support for generating a Swagger (aka. OpenAPI 2.0) document by converting the OpenAPI model
  mleclercq/endpoints@security-schemes...swagger
  This is quite a lot of code, but nothing tremendously complicated
• I then added support for OpenAPI extensions in both OpenApi 3.0 and Swagger, following the proposal presented in OpenAPI extensions are not supported #671 (comment)
  mleclercq/endpoints@swagger...openapi-extensions
• Finally I added a support for "composable middleware" as described in Be able to compose endpoints at the algebra level #794 (comment)
  mleclercq/endpoints@master...middleware

With all that, I managed to write an algebra for some google-endpoints features (gcpEndpoints/algebra/Endpoints.scala) and write a Sample API (gcpEndpoints/sample/SampleApi.scala)

I would be happy to open PRs for the these changes, but they are quite big so I'm not sure what would be the best strategy. Note that the middleware part is completely independent from the OpenApi part, so they could be reviewed/discussed separately

[julienrf]

Thank you @mleclercq for the detailed explanation! I agree that it would be very useful to support Google Endpoints.

So I tried to tackle these different issues and I managed to get something that seems to be working:

• I fixed the support for SecurityRequirement and SecurityScheme
  mleclercq/endpoints@master...security-schemes
  This also completes the support for the Info object according the the OpenAPI 3.0 spec.

I didn’t have a look at the diff, but the description you gave here makes me think that these changes should be made in endpoints4s. Do you want someone to take a closer look at what it entails, or are you already interested in making a PR?

• I added support for generating a Swagger (aka. OpenAPI 2.0) document by converting the OpenAPI model
  mleclercq/endpoints@security-schemes...swagger
  This is quite a lot of code, but nothing tremendously complicated

This one, I think it would be better if it could be kept separate from endpoints4s. Is it possible, or do you need some changes in endpoints4s? (looks like it’s “just” another interpreter, right?)

• I then added support for OpenAPI extensions in both OpenApi 3.0 and Swagger, following the proposal presented in
  OpenAPI extensions are not supported #671

I see. I must say that I’m not super happy with the fact that some parts of the documentation are defined at the point where we define endpoints, and some parts of the documentation are defined at the point where we apply the OpenAPI interpreter. I might need more time to weigh the pros and cons of this approach, but I wonder if it would be better to support OpenAPI extensions at the point where we define endpoints?

• Finally I added a support for "composable middleware" as described in Be able to compose endpoints at the algebra level #794

Nice! IIUC you already opened #847 for that? I can’t review it now, unfortunately, only next week. In the meantime, if anyone wants to help, you are welcome!

[mleclercq]

This one, I think it would be better if it could be kept separate from endpoints4s. Is it possible, or do you need some changes in endpoints4s? (looks like it’s “just” another interpreter, right?)

In my experiment, I didn't do it as it's own interpreter but instead I generate the OpenAPI 3.0 and then convert it back to OpenAPI 2.0. I think that writing another interpreter would lead to significant code duplication.

I'm fine with keeping that code outside endpoints4s. It would require to duplicate some code related to the transformation to ujson but that's fine (some classes from OpenApi 3.0 are reused in the OpenApi 2.0 model, especially the json schemas)

I see. I must say that I’m not super happy with the fact that some parts of the documentation are defined at the point where we define endpoints, and some parts of the documentation are defined at the point where we apply the OpenAPI interpreter.

Yes I agree with that concern. My take on it was that one can define a custom "high level" algebra that is used to define the endpoints and then the OpenApi interpreter for that custom algebra takes care of mapping these "high level" operations to their corresponding OpenApi extensions while the client and/or the server interpreters may implement these differently.

For instance, the algebra to add quotas on an endpoint could look like:

type WithQuota[A]

def endpointWithQuota[A, B](
    endpoint: Endpoint[A, B],
    costs: (XGoogleManagement.Metric, Int)*
): Endpoint[A, WithQuota[B]]

implicit class EndpointQuotaOps[A, B](endpoint: Endpoint[A, B]) {
  def withQuota(costs: (XGoogleManagement.Metric, Int)*): Endpoint[A, WithQuota[B]] =
    endpointWithQuota(endpoint, costs: _*)
}

Which is then used to define the endpoints as:

val myMetric: XGoogleManagement.Metric = ???

val myEndpoint: Endpoint[Unit, WithQuota[String]] =
  endpoint(
    get(path / "foo"),
    ok(emptyResponse)
  ).withQuota(myMetric -> 10)

Then the OpenApi interpreter for endpointWithQuota takes care of adding the proper OpenApi extension. And the client interpreter for that operation could implement an automatic retry with an exponential backoff.