304 response validation

[MrBananaLord]

Hi, I wonder how/if you are validating the 304 responses?

I run into an issue where response validation raised ResponseInvalid error when the rails app responded with 304. It's because this line expects the content type to be present and it's nil for the 304.

Since I am using custom middleware, I just won't validate 304's, but that might be worth looking into and thinking about :)

[ahx]

Hi. I am reworking the whole validation process so now is probably the best time to decide on the nitty gritty like this before 1.0. So thanks a lot for the message.
Currently validation fails if the API responds with a status that is not defined.
I am pretty sure that response validation should fail if the API responds with any 100-299 status which is not defined in the API description.
I am wondering if it makes sense to relax that validation in general for all 300-599 responses so that if the API responds with 300-599 we just pass/skip validation if the response is not defined.

About 304 specifically: As you may have seen response body validation is skipped for 204 but not for 304 which indeed does not make much sense and we should skip that for 304 as well.

[MrBananaLord]

Hi!
Firstly, my bad, I defined content on the 304 🤦 that's how I ended up getting an error. After properly defining 304 without content in my schema, everything worked fine.

Secondly, I enjoy the fact that 304 was picked up as missing and I think it should happen for all the codes. After all the openapi schema describes the interface so if 204 is returned from the server and it's not defined in the schema then I consider it a bug/mistake :)
To give some more context, we are adding the schema to existing endpoints and logging warnings when the request/response is invalid in order to be able to fully document the existing structure so it's beneficial to see warnings for each undefined code!

Having that said I think it would be great to allow some customisation in this area for example by adding a config for status codes to be ignored when validating (or a flag to skip validation for no-content responses). What do you think about that?

[ahx]

To give some more context, we are adding the schema to existing endpoints and logging warnings when the request/response is invalid in order to be able to fully document the existing structure

I have seen that use case before. In fact some colleagues at work do that. I am wondering if openapi_first should support that inside the middleware (or manual request/response validation), but I am not sure how the interface / options should look like.
How about adding something like on_error: ->(some_error_object) {...}, where the passed Proc would be called before the exception is raised / error response is returned. If you would combine that with raise_error: false inside the middleware or #validate instead of #validate! (#195) than your use case is covered, right?

Having that said I think it would be great to allow some customization in this area for example by adding a config for status codes to be ignored when validating (or a flag to skip validation for no-content responses). What do you think about that?

We still want to run response header validation for no-content responses. One use case I see in this area is like "I don't see value in describing 4XX responses for each operation", which is not a bad use case at all I think. I am interested in more use cases / needs here.

Firstly, my bad, I defined content on the 304 🤦 that's how I ended up getting an error. After properly defining 304 without content in my schema, everything worked fine.

I will try to improve the error message to make that part easier to follow.

[MrBananaLord]

I have seen that use case before. In fact some colleagues at work do that. I am wondering if openapi_first should support that inside the middleware (or manual request/response validation), but I am not sure how the interface / options should look like.
How about adding something like on_error: ->(some_error_object) {...}, where the passed Proc would be called before the exception is raised / error response is returned. If you would combine that with raise_error: false inside the middleware or #validate instead of #validate! (#195) than your use case is covered, right?

That would be one option... I am wondering though if that's needed. For version 1, having a clear validation interface (i.e. (#195)) and the middleware relying on the validate! should be enough.
It seems like people might want to build their own middleware to handle all the cases the way they want to (i.e. #169 (comment)). With that in mind, it seems more important to have a consistent validation behaviour than to have middleware features. Also, going this way we could keep the behaviour simpler (fewer features to maintain) while allowing a lot of flexibility if needed.
I am trying to phrase it in a way that makes sense. I hope I somehow managed to convey the idea 😅

We still want to run response header validation for no-content responses. One use case I see in this area is like "I don't see value in describing 4XX responses for each operation", which is not a bad use case at all I think. I am interested in more use cases / needs here.
My use case is as follows:

• validate only API paths (two different prefixes)
• ignore validation and do not warn on non-api paths
• warn when:
  • API path is not documented in the schema
  • response is invalid (e.g. status, body, content type, etc; I'd expect to be notified about 304 and 204 not being defined in the schema)
  • request it invalid and given path is marked as "soft validate" (when documenting the existing request we need to have a grace period)
• respond with validation error when the request is invalid and the given path is not in the grace period anymore

I handle all that in my custom validator class which relies on the request and response validation interface. Since it's a lot of custom requirements I think it's best handled this way.

Following that, I think that for now, it's not worth generalizing the middleware behaviour too much (already mentioned in the previous paragraph) because it seems like there are too many ways people might want to set it up. Instead, having a simple default middleware and a simple and consistent validation layer would do the trick :)

[ahx]

Thanks for the reply @MrBananaLord. I think I understand your use case better now. And I think I understand the need for building a custom middleware.

validate only API paths (two different prefixes)

Are you aware of the only: option of OpenapiFirst.load? It's one way to fully filter out certain path once during initialization so you don't have to skip things during request time.

[MrBananaLord]

Yes, I am aware of that feature :) it's not exactly what I was looking for because I want to log a warning if the path is not defined in the schema. For example with the request "GET /foo" I want to entirely ignore it (skip validation) and for "GET /api/foo" I want to raise a warning when this path is not in my schema. This is not really related to this gem but rather to the process we are following to ensure we document all the endpoints.

[ahx]

@MrBananaLord is this resolved or are there still some open ends?

[MrBananaLord]

yes, it's resolved! thank you :)