Enable prepared statements over http.client._MAXLINE limit #173
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hovaesco
reviewed
Apr 26, 2022
| @@ -9,6 +9,7 @@ | |||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | |||
| # See the License for the specific language governing permissions and | |||
| # limitations under the License. | |||
| import http | |||
There was a problem hiding this comment.
from http.client import _MAXLINE
There was a problem hiding this comment.
That actually doesn't comply with the pre-commit check as it doesn't support private attributes.
tests/integration/test_dbapi_integration.py:15: error: Module "http.client" has no attribute "_MAXLINE"
Found 1 error in 1 file (checked 32 source files)
I think we don't want to import http.client at all in the module namespace as we only rely on some private property. I moved the import to the test itself.
trino/client.py
Outdated
| headers.append(line) | ||
| if line in (b'\r\n', b'\n', b''): | ||
| break | ||
| hstring = b''.join(headers).decode('iso-8859-1') |
trino/client.py
Outdated
|
|
||
|
|
||
| # Disable header limit to avoid LineTooLong('got more than 65536 bytes when reading header line') | ||
| disable_header_limit() |
There was a problem hiding this comment.
Could this be moved into e.g ClientSession or TrinoRequest class?
There was a problem hiding this comment.
It feels a lot less pythonic to me. it works of course and hides this override in TrinoRequest instead of the module's namespace.
class TrinoRequest:
@staticmethod
def _disable_header_limit():
import http.client
import email.parser
from http.client import HTTPMessage, _MAXLINE
def parse_headers(fp, _class=HTTPMessage):
headers = []
while True:
line = fp.readline(_MAXLINE + 1)
headers.append(line)
if line in (b'\r\n', b'\n', b''):
break
header_string = b''.join(headers).decode('iso-8859-1')
return email.parser.Parser(_class=_class).parsestr(header_string)
http.client.parse_headers = parse_headers
# Disable header limit to avoid LineTooLong('got more than 65536 bytes when reading header line')
_disable_header_limit.__func__()
There was a problem hiding this comment.
I created a helper class for this override. Let me know WDYT.
mdesmet
force-pushed
the
feature/fix_http_client_maxline
branch
2 times, most recently
from
d08c906
to
e3ecce6
Compare
mdesmet
force-pushed
the
feature/fix_http_client_maxline
branch
from
e3ecce6
to
00dd234
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #148
Headers can become much longer then what is currently suported in Python's
http.client. This code patches the header parsing fromhttp.clientto be able to parse the headers coming back from the Trino server.Test is included. when removing the
disable_header_limit()in client.py, the test will fail withrequests.exceptions.ConnectionError: ('Connection aborted.', LineTooLong('got more than 65536 bytes when reading header line')).