openapi: fix import of big files

[megalucio]

Overview

Fix import openapi definitions from files. It fails with big files.

Related Issues

Specify any related issues or pull requests by linking to them.

zaproxy/zaproxy#8193

Checklist

• Update help
• Update changelog
• Run ./gradlew spotlessApply for code formatting
• Write tests
• Check code coverage
• Sign-off commits
• Squash commits
• Use a descriptive title

For more details, please refer to the developer rules and guidelines.

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[megalucio]

recheck

[thc202]

You first need to add the agreeing comment.

[megalucio]

I have read the CLA Document and I hereby sign the CLA

[thc202]

It's this way to add support for multi file definitions, #2913, doing just this change breaks that.

[megalucio]

Any good examples to test this multi file definition schema? I tried with https://github.com/Paldom/swagger-multifile-example/tree/master and in any case this does not seem to work either, getting different errors.

[thc202]

The codebase has tests for that, ExtensionOpenApiTest.shouldImportMultiFileV*.

[thc202]

IMO this is not the proper fix, we should change the parser limits instead.

[thc202]

This checks will not work for JSON format (in that it will still fail to parse big files).

[megalucio]

IMO this is not the proper fix, we should change the parser limits instead.

Agreed, however I understand the limit that is being reached here is the String's Maximum length(Integer.MAX_INT or 2147483647) as seen on the exception

com.fasterxml.jackson.databind.JsonMappingException: TextBuffer overrun: size reached (2147483648)
...
Caused by: java.lang.IllegalStateException: TextBuffer overrun: size reached (2147483648)
...

Not sure how to address that. Besides the method where the exception is being thrown, Json.pretty() belongs to external library swagger-core so it does not look like we can change that.

Another alternative which I was thinking was to catch for the IllegalStateException and only on that instance to get the string directly from the file, which seems a bit more proper.

Thoughts? Any other ideas on how to fix it?

[thc202]

It looks suspicious that's actually hitting the max of the string, what's the size of the original definition? Did you try write the expanded definition to a file to see its size (and contents for that matter)?

I'm aware that's a class/method from the library, but that's "one" line of code.

Agreed that looks better still I'd skip the pretty print instead of reading from the file (as that would break the multi file).

[kingthorin]

@megalucio are you able to answer the latest questions and keep going with this?