StackOverflowError on AbstractRequestHelper.makeRequest

[catsncode]

Hey there, thanks for the great API. It works sometimes for me, but will sometimes stop working and throw this StackOverflowError

...
at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:130)
at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:130)
at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:130)
....

I am using version 2.23

It's possible we are hitting some kind of twitter rate limit, but it looks like the limit is 300 post tweets in 3 hours, which we are definitely not doing. I would say at most we have done 50 or so in 3 hours.

Thanks for any help!

[redouane59]

Hello,
Can you check if you have other error message and share the full stack trace ? Normally if you reached the limit, you have a explicite output telling it.

[catsncode]

Thanks for the help and sorry for the delay, for some reason I didn't get an email from github. I'm happy to try a debug jar or anything else to help fix this.

I do not see any errors from Twitter. Our account says "Tweets - post up to 1500 Tweets per month" and I think the rate limit is 300 per 3 hours, neither of which we are coming close to.

Here is the full log entry:

13-Sep-2023 07:48:11.610 SEVERE [https-openssl-apr-443-exec-1120] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [action] in context with path [] threw exception [Servlet execution threw an exception] with root cause
java.lang.StackOverflowError
at java.base/sun.nio.ch.NativeThread.current(NativeThread.java:51)
at java.base/sun.nio.ch.NioSocketImpl.beginRead(NioSocketImpl.java:222)
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:289)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:483)
at java.base/sun.security.ssl.SSLSocketInputRecord.readFully(SSLSocketInputRecord.java:466)
at java.base/sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:243)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:181)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1513)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1484)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310)
at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361)
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:827)
at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:759)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1684)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1585)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:529)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:308)
at com.github.scribejava.core.httpclient.jdk.JDKHttpClient.doExecute(JDKHttpClient.java:134)
at com.github.scribejava.core.httpclient.jdk.JDKHttpClient.execute(JDKHttpClient.java:95)
at com.github.scribejava.core.oauth.OAuthService.execute(OAuthService.java:114)
at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:100)
at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:130)
at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:130)
at io.github.redouane59.twitter.helpers.AbstractRequestHelper.makeRequest(AbstractRequestHelper.java:130)
(repeats 638 times)

            at io.github.redouane59.twitter.helpers.RequestHelper.postRequestWithBodyJson(RequestHelper.java:27)
            at io.github.redouane59.twitter.TwitterClient.postTweet(TwitterClient.java:793)
            at io.github.redouane59.twitter.TwitterClient.postTweet(TwitterClient.java:785)
            at com.portal.util.TweetSender.sendTweet(TweetSender.java:171)

[redouane59]

Hmmm there is no so much logs related to the library itself. Are you able to provoke the error on purpose ? Did you try to debug ?
I'm a bit stuck because I never saw it before, the StackOverflowError should not be related to an exceeded limit, which has his own exception.

[catsncode]

I'm not sure if it helps, but it seems like when this happens, it takes around 15 minutes before returning that error. So I guess it's just callilng that AbstractRequestHelper.makeRequest over and over for that long.

I could try to debug it in VS code, but it seems like it would just show me the same info as the stack?

[catsncode]

To also give some context, yesterday we were live tweeting a news conference and the code was working fine for around 15 tweets over a period of 20 minutes, then all of a sudden it starts giving this error and it won't work again for maybe a day.

[redouane59]

I could try to debug it in VS code, but it seems like it would just show me the same info as the stack?

if you could fork the project and run your code locally, you will be able to debug step by step in the lib itself.

But what you say is really strange, are you sure that you don't call the methods too much, like in an infinite loop ?

[catsncode]

ok maybe I will try that.

Yes I'm sure my code is only being run once. As you can see in the stack trace, my code is only run once at the very bottom (com.portal)

at com.portal.util.TweetSender.sendTweet(TweetSender.java:171)

and my tomcat log is only showing this error once per tweet attempt.

[catsncode]

I'm finally getting around to tracing through this, but having some compile problems.

There is a LOGGER defined in a bunch of files, but the @slf4j annotation only provides "log", not LOGGER. Sorry if I am missing something obvious and thanks again for the help.

[redouane59]

Check this config file : https://github.com/redouane59/twittered/blob/develop/lombok.config

[catsncode]

Thanks, that fixed it.

I realized I'm using an older version of the scribe libraries, slf4j, and lombok, so I'm thinking one of those may be the cause of my issue. I've got everything upgraded and will test it out tonight.

Thanks again.

[catsncode]

well I upgraded all the scribe and slf libraries to the versions you specified, but still seeing the stack overflow in the same place. I'm going to try and trace it tonight.

On the twitter developer page where it has the keys, am I correct that I do not need to use the OAuth 2.0 client ID and client secret?

Just the consumer keys and access token and secret right?

I assume so since the code was working sometimes, but just making sure.

Thanks!

[catsncode]

Ok I think I figured out the issue. The headers say my account and app have 0 tweets remaining, which makes no sense since the API hasn't been working. Even if it had been, we would have only posted 10 or so tweets in the past 24 hours, but regardless that is what the headers say. So I think the code below is infinitely retrying and eventually throws a stack overflow. Just needs some kind of limit I guess, 100 tries or whatever makes sense.

Do you have any idea what the 24hour reset time is? Even if that value is milliseconds, that's a long time. I'm not sure if it's a twitter bug or it represents some kind of API ban. It shouldn't as everything looks good on the twitter developer page and we haven't done anything out of the ordinary.

AbstractRequestHelper

public Optional makeRequest(OAuthRequest request, boolean signRequired, Class classType) {
...
String stringResponse = response.getBody();
LOGGER.debug("Response code: {} to url: '{}' headers: x-rate-limit-reset: {} x-rate-limit-remaining: {}", response.getCode(), request.getUrl(), response.getHeader("x-rate-limit-reset"), response.getHeader("x-rate-limit-remaining"));
....
return makeRequest(request, false, classType); // We have already signed if it was requested

Header values from the response object:

HashMap$Node@211 "x-app-limit-24hour-remaining":"0"
HashMap$Node@213 "x-user-limit-24hour-remaining":"0"
HashMap$Node@214 "x-user-limit-24hour-reset":"1695487207"
HashMap$Node@221 "x-app-limit-24hour-reset":"1695496641"
HashMap$Node@222 "x-rate-limit-reset":"1695442714"
HashMap$Node@223 "x-rate-limit-limit":"40000"
HashMap$Node@224 "x-rate-limit-remaining":"39998"
HashMap$Node@225 "x-app-limit-24hour-limit":"50"
HashMap$Node@231 "x-user-limit-24hour-limit":"50"

Thanks for any help!

[redouane59]

Hmmm I never saw this before, maybe you can try to post on the twitter dev forum (if it still exist). It's possible that on the API side they changed something that make this problem appears recently (for example, before, doing a try didn't consume the call quota).
As Twitter have shut down all the dev community programs, I just stop being as involved as before regarding this library evolution and support so I'm a bit limited to help you directly.

[catsncode]

Thanks for the reply. I did post on the twitter dev forum, but I'm not too hopeful.

It sounds like you are saying there is a possibility that the API is somehow consuming all 50 tweets, without actually posting one?

I will just see if I can find a different library since you are not actively working on this anymore. Any suggestions? Maybe I can just use scribe for the oauth and hit the api directly? For some reason I am finding this V2 twitter API much more confusing than version 1.

Thanks again!

[redouane59]

It sounds like you are saying there is a possibility that the API is somehow consuming all 50 tweets, without actually posting one?

In fact what happens is that when you call an endpoint, in the case that you reached a limit, the code will wait for 5min before doing another try. Normally, the try (without positive result) doesn't call as a consumption.

I don't know if any java library exist, with what happened 8 months ago, I think that everybody decided to give up.

But yes you can totally develop the code on your side to call the API, copying what interests you.

[catsncode]

Roger that, thanks a lot and take care!