API stopped working with struct.error: unpack requires a buffer of 1 bytes #27

alekthebear · 2020-06-04T04:24:09Z

I've had a long running job using the evernote python 3 API. It suddenly stopped working today. I am seeing the following errors:

Traceback (most recent call last):
  File "EDAMTest.py", line 49, in <module>
    UserStoreConstants.EDAM_VERSION_MINOR
  File "/Users/xxxxx/Applications/miniconda3/envs/xxxxx/lib/python3.7/site-packages/evernote/api/client.py", line 161, in delegate_method
    return targetMethod(*args, **kwargs)
  File "/Users/xxxxx/Applications/miniconda3/envs/xxxxx/lib/python3.7/site-packages/evernote/edam/userstore/UserStore.py", line 491, in checkVersion
    return self.recv_checkVersion()
  File "/Users/xxxxx/Applications/miniconda3/envs/xxxxx/lib/python3.7/site-packages/evernote/edam/userstore/UserStore.py", line 504, in recv_checkVersion
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "/Users/xxxxx/Applications/miniconda3/envs/xxxxx/lib/python3.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 145, in readMessageBegin
    type = self.readByte()
  File "/Users/xxxxx/Applications/miniconda3/envs/xxxxx/lib/python3.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 201, in readByte
    val, = unpack('!b', buff)
struct.error: unpack requires a buffer of 1 bytes

Digging around random stack traces show that :

> /Users/xxxxx/Applications/miniconda3/envs/xxxxx/lib/python3.7/site-packages/thrift/protocol/TBinaryProtocol.py(145)readMessageBegin()
-> type = self.readByte()
(Pdb) l
140         else:
141           if self.strictRead:
142             raise TProtocolException(type=TProtocolException.BAD_VERSION,
143                                      message='No protocol version header')
144           name = self.trans.readAll(sz)
145  ->       type = self.readByte()
(Pdb) name
b'ctype html><html lang="en"><head><title>HTTP Status 400 \xe2\x80\x93 Bad Request</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 400 \xe2\x80\x93 Bad Request</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Description</b> The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).</p><hr class="line" /><h3>Apache Tomcat/8.5.54</h3></body></html>'

Is the API no longer worker or deprecated?

The text was updated successfully, but these errors were encountered:

ooke · 2020-06-04T08:56:27Z

I have the same problem, it just stopped working today, reproducible for me with python3.7 and python3.8.

mitchelljustin · 2020-06-04T17:39:28Z

I think i've found the reason for this error. My hypothesis is that Evernote recently updated their API to throw an error for HTTP requests with duplicate Host headers. As it turns out, this Python 3 package duplicates the Host header for some requests. The result is this really obscure error.

When I monkey patch the Thrift HTTP code to dedupe Host headers, my code works again.

mitchelljustin · 2020-06-04T17:54:40Z

Found out it's because this SDK uses a (really) old version of thrift. The duplicate host problem in Thrift was seemingly fixed in apache/thrift@760511f#comments, 3.5 years ago

heart5 · 2020-06-04T18:33:37Z

I think i've found the reason for this error. My hypothesis is that Evernote recently updated their API to throw an error for HTTP requests with duplicate Host headers. As it turns out, this Python 3 package duplicates the Host header for some requests. The result is this really obscure error.

When I monkey patch the Thrift HTTP code to dedupe Host headers, my code works again.

How can I fixed it like yours. Thank you very much.

mitchelljustin · 2020-06-04T18:34:43Z

Here's a gist with the monkey patch https://gist.github.com/mitchelljustin/46e5ce3e0ddfb2ee154ac9a6ee696ac1

Note, to be proper this SDK should be forked and updated to a later version of Thrift. I tried that but got all kinds of other errors I didn't want to fix. Unfortunately Evernote doesn't maintain this repo anymore.

iafan · 2020-06-05T00:52:26Z

This is Evernote speaking. Thanks for reporting this issue! Our engineers are working on a solution, and we will report here when the update is available.

… (fixes 216) Workaround proposed here Evernote/evernote-sdk-python3#27

zonabi · 2020-06-10T23:08:10Z

Hey everyone. Thanks for your patience and info. (@mitchelljustin was on the money) We have just merged a patch for this, please let me know that it solves the issue or if any calls are still failing.

heart5 · 2020-06-11T03:33:13Z

Hey everyone. Thanks for your patience and info. (@mitchelljustin was on the money) We have just merged a patch for this, please let me know that it solves the issue or if any calls are still failing.

It works fine, great. Thanks for members.

KostyaEsmukov · 2020-06-11T06:25:15Z

@zonabi Thanks for the merge! Would it be possible to make a new release on pypi?

zonabi · 2020-06-15T18:46:35Z

@KostyaEsmukov a new version of the evernote3 package has been released on PyPI (1.25.14)

Let us know if you run into any more issues.

KostyaEsmukov · 2020-06-15T20:58:36Z

@zonabi Perfect, thank you! I can confirm that the uploaded version works.

Update evernote-sdk-python3 to 1.25.14 to fix Evernote/evernote-sdk-python3#27

TristanH · 2021-04-13T01:05:58Z

Hiya @zonabi @iafan this issue seems to be happening once again, even on the new version of this package

Notably, it is not happening in sandbox mode, but IS happening in production.

In case it's useful, we have thousands of users who depend on our integration with Evernote, and right now it's completely broken for all of them.

TristanH · 2021-04-13T01:25:51Z

Here's a stack trace:

  File "home/main/integrations/evernote.py", line 35, in prepare_export
    self.note_store = self.client.get_note_store()
  File "home/venv/lib/python3.8/site-packages/evernote/api/client.py", line 106, in get_note_store
    note_store_uri = user_store.getNoteStoreUrl()
  File "home/venv/lib/python3.8/site-packages/evernote/api/client.py", line 165, in delegate_method
    return functools.partial(
  File "home/venv/lib/python3.8/site-packages/evernote/edam/userstore/UserStore.py", line 1156, in getNoteStoreUrl
    return self.recv_getNoteStoreUrl()
  File "home/venv/lib/python3.8/site-packages/evernote/edam/userstore/UserStore.py", line 1167, in recv_getNoteStoreUrl
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "home/venv/lib/python3.8/site-packages/thrift/protocol/TBinaryProtocol.py", line 145, in readMessageBegin
    type = self.readByte()
  File "home/venv/lib/python3.8/site-packages/thrift/protocol/TBinaryProtocol.py", line 201, in readByte
    val, = unpack('!b', buff)
struct.error: unpack requires a buffer of 1 bytes

TristanH · 2021-04-13T04:01:12Z

Looks like the getNoteStoreUrl requests are returning this error:

L><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http&#58;&#47;&#47;www&#46;evernote&#46;com&#47;edam&#47;user" on this server.<P>\nReference&#32;&#35;18&#46;7937b9d0&#46;1618283633&#46;f8c4dd1\n</BODY>\n</HTML>\n

This request is the same as it was yesterday, but broken now -- so probably a change to the evernote api to improperly return Access Denied?

heart5 · 2021-04-13T10:26:22Z

Hiya @zonabi @iafan this issue seems to be happening once again, even on the new version of this package

Notably, it is not happening in sandbox mode, but IS happening in production.

In case it's useful, we have thousands of users who depend on our integration with Evernote, and right now it's completely broken for all of them.

It's seems sites have change the permission for API

duguguiyu · 2021-04-13T11:19:00Z

The same error, when i call getNoteStoreUrl

rdhyee · 2021-04-13T14:37:52Z

I'm seeing the same problem this morning. Trying to get the attention of Evernote Support: https://twitter.com/rdhyee/status/1381977939072311297

I just reported the issue with Evernote -- it's been filed as Ticket# 3296044

zonabi · 2021-04-13T15:11:56Z

Evernote engineer here, acknowledging these issues. We are looking into it.

TristanH · 2021-04-13T19:36:30Z

@zonabi some other stuff from my debugging, in case it's useful:

The javascript sdk doesn't seem to have this issue, but the python2 sdk does
other requests on UserStore like checkVersion are broken, but requests to NoteStore like getNotebook and createNote seem to be working fine

zonabi · 2021-04-13T20:33:58Z

@TristanH @heart5 @rdhyee @duguguiyu the issue should be resolved now, please let us know if that is not the case!

rdhyee · 2021-04-13T20:46:08Z

@zonabi My application works again -- thank you!

TristanH · 2021-04-13T21:13:37Z

@zonabi yes! Seems to be fixed. Thanks for the quick turnaround :)))

alekthebear · 2023-01-08T15:10:21Z

Update: Fixed as of 1/10/23

Running into the same issue as #27 (comment) (Access Denied) once again

Stacktrace:

  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/evernote/api/client.py", line 106, in get_note_store
    note_store_uri = user_store.getNoteStoreUrl()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/evernote/api/client.py", line 165, in delegate_method
    return functools.partial(
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/evernote/edam/userstore/UserStore.py", line 1156, in getNoteStoreUrl
    return self.recv_getNoteStoreUrl()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/evernote/edam/userstore/UserStore.py", line 1167, in recv_getNoteStoreUrl
    (fname, mtype, rseqid) = self._iprot.readMessageBegin()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/thrift/protocol/TBinaryProtocol.py", line 145, in readMessageBegin
    type = self.readByte()
  File "/layers/google.python.pip/pip/lib/python3.9/site-packages/thrift/protocol/TBinaryProtocol.py", line 201, in readByte
    val, = unpack('!b', buff)
struct.error: unpack requires a buffer of 1 bytes

Debug info:

> /Users/xxxxx/Applications/miniconda3/envs/xxxxx/lib/python3.9/site-packages/evernote/edam/userstore/UserStore.py(1167)recv_getNoteStoreUrl()
   1165 
   1166   def recv_getNoteStoreUrl(self, ):
-> 1167     (fname, mtype, rseqid) = self._iprot.readMessageBegin()
   1168     if mtype == TMessageType.EXCEPTION:
   1169       x = TApplicationException()

ipdb> d
> /Users/xxxxx/Applications/miniconda3/envs/xxxxx/lib/python3.9/site-packages/thrift/protocol/TBinaryProtocol.py(145)readMessageBegin()
    143                                  message='No protocol version header')
    144       name = self.trans.readAll(sz)
--> 145       type = self.readByte()
    146       seqid = self.readI32()
    147     return (name, type, seqid)

ipdb> l
    140     else:
    141       if self.strictRead:
    142         raise TProtocolException(type=TProtocolException.BAD_VERSION,
    143                                  message='No protocol version header')
    144       name = self.trans.readAll(sz)
--> 145       type = self.readByte()
    146       seqid = self.readI32()
    147     return (name, type, seqid)
    148 
    149   def readMessageEnd(self):
    150     pass

ipdb> name
b'L><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http&#58;&#47;&#47;www&#46;evernote&#46;com&#47;edam&#47;user" on this server.<P>\nReference&#32;&#35;18&#46;2dca4d17&#46;1673189791&#46;f0dd35d4\n</BODY>\n</HTML>\n'

manjurulhoque · 2023-10-04T18:21:40Z

Facing the same issue again.

bordaigorl mentioned this issue Jun 5, 2020

Unpack requires a bytes object of length 1 bordaigorl/sublime-evernote#216

Closed

bordaigorl added a commit to bordaigorl/sublime-evernote that referenced this issue Jun 5, 2020

Fixing SDK, API returning "Bad Request" due to duplicate host headers…

cdc1491

… (fixes 216) Workaround proposed here Evernote/evernote-sdk-python3#27

zonabi mentioned this issue Jun 5, 2020

Temp fix for duplicate header failures #28

Merged

AuroraDysis mentioned this issue Jun 18, 2020

No any text in Evernote if TG message contains photo djudman/evernote-telegram-bot#25

Closed

AuroraDysis added a commit to AuroraDysis/evernote-telegram-bot that referenced this issue Jun 18, 2020

Update evernote-sdk-python3 to 1.25.14

259e09b

Update evernote-sdk-python3 to 1.25.14 to fix Evernote/evernote-sdk-python3#27

AuroraDysis mentioned this issue Jun 18, 2020

Update evernote-sdk-python3 to 1.25.14 djudman/evernote-telegram-bot#28

Merged

felciano mentioned this issue Sep 4, 2021

Occasionally getting struct.error exception on fresh sync vzhd1701/evernote-backup#3

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API stopped working with struct.error: unpack requires a buffer of 1 bytes #27

API stopped working with struct.error: unpack requires a buffer of 1 bytes #27

alekthebear commented Jun 4, 2020

ooke commented Jun 4, 2020

mitchelljustin commented Jun 4, 2020 •

edited

mitchelljustin commented Jun 4, 2020

heart5 commented Jun 4, 2020

mitchelljustin commented Jun 4, 2020 •

edited

iafan commented Jun 5, 2020

zonabi commented Jun 10, 2020

heart5 commented Jun 11, 2020

KostyaEsmukov commented Jun 11, 2020

zonabi commented Jun 15, 2020

KostyaEsmukov commented Jun 15, 2020

TristanH commented Apr 13, 2021 •

edited

TristanH commented Apr 13, 2021 •

edited

TristanH commented Apr 13, 2021

heart5 commented Apr 13, 2021

duguguiyu commented Apr 13, 2021

rdhyee commented Apr 13, 2021 •

edited

zonabi commented Apr 13, 2021

TristanH commented Apr 13, 2021

zonabi commented Apr 13, 2021

rdhyee commented Apr 13, 2021

TristanH commented Apr 13, 2021

alekthebear commented Jan 8, 2023 •

edited

manjurulhoque commented Oct 4, 2023

API stopped working with struct.error: unpack requires a buffer of 1 bytes #27

API stopped working with struct.error: unpack requires a buffer of 1 bytes #27

Comments

alekthebear commented Jun 4, 2020

ooke commented Jun 4, 2020

mitchelljustin commented Jun 4, 2020 • edited

mitchelljustin commented Jun 4, 2020

heart5 commented Jun 4, 2020

mitchelljustin commented Jun 4, 2020 • edited

iafan commented Jun 5, 2020

zonabi commented Jun 10, 2020

heart5 commented Jun 11, 2020

KostyaEsmukov commented Jun 11, 2020

zonabi commented Jun 15, 2020

KostyaEsmukov commented Jun 15, 2020

TristanH commented Apr 13, 2021 • edited

TristanH commented Apr 13, 2021 • edited

TristanH commented Apr 13, 2021

heart5 commented Apr 13, 2021

duguguiyu commented Apr 13, 2021

rdhyee commented Apr 13, 2021 • edited

zonabi commented Apr 13, 2021

TristanH commented Apr 13, 2021

zonabi commented Apr 13, 2021

rdhyee commented Apr 13, 2021

TristanH commented Apr 13, 2021

alekthebear commented Jan 8, 2023 • edited

manjurulhoque commented Oct 4, 2023

mitchelljustin commented Jun 4, 2020 •

edited

mitchelljustin commented Jun 4, 2020 •

edited

TristanH commented Apr 13, 2021 •

edited

TristanH commented Apr 13, 2021 •

edited

rdhyee commented Apr 13, 2021 •

edited

alekthebear commented Jan 8, 2023 •

edited