Tweepy cursor not reaching its limit

[Lueesla]

I am coding a twitter bot which join giveaways of users that I follow.

The problem is that when I use a for loop to iterate a ItemIterator Cursor of 50 items it breaks before finishing. It usually does 20 or 39-40 iterations.

My main function is:

from funciones import *
from config import *
api = login(user)
i=0
while 1>i:
    tweets = get_tweets(api, 50, True, None, None)
    file = start_stats()
    for tweet in tweets:
      try:
          i = i+1
          tweet = is_RT(tweet)
          show(tweet)
          check(api,tweet,file)
          print(f'{i}) 1.5 - 2m tweets cd')
          sleep(random.randrange(40, 60,1))
      except Exception as e:
          print(str(e))
          st.append(e)
    print('15-20 min cooldown')
    sleep(random.randrange(900, 1200,1))

So when the loop do usually 39 iterations, the code jumps into the 15min cooldown getting these of tweets:

len(tweets.current_page) - 1
Out[251]: 19

tweets.page_index
Out[252]: 19

tweets.limit
Out[253]: 50

tweets.num_tweets
Out[254]: 20

I've seen this in the tweepy cursor.py but I still don't know how to fix it.

  def next(self):
        if self.limit > 0:
            if self.num_tweets == self.limit:
                raise StopIteration
        if self.current_page is None or self.page_index == len(self.current_page) - 1:
            # Reached end of current page, get the next page...
            self.current_page = self.page_iterator.next()
            self.page_index = -1
        self.page_index += 1
        self.num_tweets += 1
        return self.current_page[self.page_index]
The function I use in my main function to get the cursor is this:

def get_tweets(api,count=1,cursor = False, user = None, id = None):
    if id is not None:
        tweets = api.get_status(id=id, tweet_mode='extended')
        return tweets
    
    if cursor:
        if user is not None:
            if count>0:
                tweets = tp.Cursor(api.user_timeline, screen_name=user, tweet_mode='extended').items(count)
            else:
                tweets = tp.Cursor(api.user_timeline, screen_name=user, tweet_mode='extended').items()
        else:
            if count>0:
                tweets = tp.Cursor(api.home_timeline, tweet_mode='extended').items(count)
            else:
                tweets = tp.Cursor(api.home_timeline, tweet_mode='extended').items()
    else:
        if user is not None:
            tweets = api.user_timeline(screen_name=user, count=count,tweet_mode='extended')
        else:
            tweets = api.home_timeline(count=count, tweet_mode='extended')
    return tweets

When I've tried test codes like

j = 0
tweets = get_tweets(api,100,True)
for i in tweets:
    j=j+1
print(j)

j and tweets.num_tweets are almost always 100, but I think when this is not 50 is because I dont wait between request, because I've reached j=300 with this, so maybe the problem is in the check function:

(It's a previous check function which also has the same problem, I've noticed it when I've started getting stats, the only difference is that I return values if the tweets has been liked, rt, etc)

def check(tweet):
    if (bool(is_seen(tweet))
     +  bool(age_check(tweet,3))
     +  bool(ignore_check(tweet)) == 0):
        rt_check(tweet)
        like_check(tweet)
        follow_check(tweet)
        tag_n_drop_check(tweet)
        quoted_check(tweet)

This is the first time I asked help so I dont know if I've post all the info needed, this is driving me mad since last week and I dont know who to ask :(

Thanks in advance!

[Harmon758]

Is there an error? If so, what's the full traceback?
It's difficult to tell what exactly is going on without all of the relevant code.

[Lueesla]

There is no error, I can paste the full code, i thought it was not important because I am not changing the cursor, just the element

[Harmon758]

I'm not sure what you mean by that, and I'm not clear on how you're determining that you're not receiving the expected number of Tweets.

[Lueesla]

Once the main does a full loop (it reachs the last sleep) the cursor called 'tweets' returns these values:

tweets.limit
Out[253]: 50

tweets.num_tweets
Out[254]: 20

It doesnt reach the limit, and I am also checking it with a variable that adds 1 each loop.

Im using api.home_timeline with count = 100 and it works perfectly

[Lueesla]

def is_seen(tweet):
    if tweet.favorited or tweet.retweeted:
        print(f"{config.bcolors.WARNING}Ya interactuado{config.bcolors.ENDC}")
        return True 
    else:
        print(f"{config.bcolors.OKGREEN}Tweet con el que no se ha interactuado{config.bcolors.ENDC}")
        return False


def ignore_check(tweet):
    words = to_words(tweet)
    if any(x in words for x in config.ban):
        print(f"{config.bcolors.WARNING}Ignorado{config.bcolors.ENDC}")
        return True
    else:
        print(f"{config.bcolors.OKGREEN}No se ignora{config.bcolors.ENDC}")
        return False
        

def rt_check(api,tweet):
    try:
        words = to_words(tweet)
        if any(x in words for x in config.rt):
            api.retweet(tweet.id)
            print(f"{config.bcolors.OKCYAN}Retweeted{config.bcolors.ENDC}")
            sleep(random.randrange(3, 8, 1))
            return True
        else:
             print('No se hace retweet')
             return False
    except Exception as e:
        print('Error no se ha podido retweetear')
        print(str(e))

def like_check(api,tweet):
    try:
        words = to_words(tweet)
        if any(x in words for x in config.fav):
            api.create_favorite(tweet.id)
            print(f"{config.bcolors.OKCYAN}Liked{config.bcolors.ENDC}")
            sleep(random.randrange(10, 15, 1))
            return True
        else:
            print('No Liked')
            return False
    except Exception as e:
        print('Error no se ha podido likear')
        print(str(e))


def follow_check(api,tweet):
    try:
        words = to_words(tweet)
        follows = []
        if any(x in words for x in config.follow):
            if tweet.author.following is False:
                api.create_friendship(screen_name=tweet.user.screen_name)
                print(f"{config.bcolors.OKCYAN}Seguido: @%s{config.bcolors.ENDC}" % tweet.user.screen_name)
                follows.append(tweet.user.screen_name)
                sleep(random.randrange(45, 60,1))
            else:
                print(f"{config.bcolors.OKCYAN}Ya seguia a: @%s{config.bcolors.ENDC}" % tweet.user.screen_name)
            for i in range(len(tweet.entities['user_mentions'])):
                ment = api.get_user(user_id=tweet.entities['user_mentions'][i].get('id'))
                if ment.following is False:
                    api.create_friendship(screen_name=ment.screen_name)
                    print(f"{config.bcolors.OKCYAN}Seguido: @%s{config.bcolors.ENDC}" % ment.screen_name)
                    follows.append(ment.screen_name)
                    sleep(random.randrange(45, 60,1))
                else:
                    print(f"{config.bcolors.OKCYAN}Ya seguia a: @%s{config.bcolors.ENDC}" % ment.screen_name)
        else:
            print('No follow')
        return follows
    except Exception as e:
        print('Error no se ha podido seguir')
        print(str(e))

def quoted_check(api,tweet):
    try:
        if hasattr(tweet, 'quoted_status_id'):
            quoted = api.get_status(tweet.quoted_status_id, tweet_mode='extended')
            print('\n')
            print('Quoted Tweet: ')
            show(quoted)
            check(api,quoted)
            quoted = api.get_status(tweet.quoted_status_id, tweet_mode='extended')
            if quoted.favorited is False:
                api.create_favorite(quoted.id)
            if quoted.retweeted is False:
                api.retweet(quoted.id)
            return True
        else:
            return False
    except Exception as e:
        print(str(e))
           
def age_check(tweet, dias=3):
    try:
        now = datetime.now(timezone.utc) 
        tvec = [0,0,0,0]
        isFound = False
        for i, unidad in enumerate(config.timeword):
            for palabra in config.timeword[unidad]:
                tvec[i] = re.findall("(\d+) ?%s(\W)? " % palabra, tweet.full_text.lower())
                if tvec[i] ==[]:
                    tvec[i] = re.findall("(\d+) ?%s(\W)?[\n]" % palabra, tweet.full_text.lower())
                if tvec[i] ==[]:
                    tvec[i] = re.findall("(\d+) ?%s(\W)?$" % palabra, tweet.full_text.lower())                  
                if tvec[i] != []:
                    for x in tvec[i]:
                        tvec[i] = int(x[0])
                    print('\t Se ha encontrado %s %s' % (tvec[i],palabra))
                    isFound = True
                    break
                else:
                    tvec[i] = 0
        if isFound:
            dt = timedelta(hours=tvec[0], minutes=tvec[1],seconds=tvec[2], days=tvec[3])
        else:
            dt = timedelta(days=dias)
        if hasattr(tweet, 'retweeted_status'):
            dif = now - tweet.retweeted_status.created_at
        else:
            dif = now - tweet.created_at
        if dif > dt:
            print(f"{config.bcolors.WARNING}Tweet muy antiguo{config.bcolors.ENDC}")
            return True
        else:
            print(f"{config.bcolors.OKGREEN}Tweet reciente{config.bcolors.ENDC}")
            return False
    except Exception as e:
        print('Error age check')
        print(str(e))

def is_RT(tweet):
    if hasattr(tweet, 'retweeted_status'):
        tweet = tweet.retweeted_status
    return tweet

def tag_n_drop_check(api,tweet):
    words = to_words(tweet)
    tg = False
    accion = False
    wltdrop = False
    try:
        if any(x in words for x in config.tag):
            tg = True
        
        for add in config.wallet:
            drop = re.findall("drop your [$,#]?(\w+) %s" % add,tweet.full_text.lower())
            if drop == []:
               drop = re.findall("drop [$,#]?(\w+) %s" % add,tweet.full_text.lower())
            if drop != []:
                if drop[0] == 'sol':
                    wltdrop = 1
                else:
                    wltdrop = 0
                break
        user = api.verify_credentials().screen_name    
        if tg and wltdrop is not False:
            api.update_status('@%s %s %s' % (tweet.author.screen_name, config.crypto[user][wltdrop], config.crew[user]), in_reply_to_status_id=tweet.id)
            print(f"{config.bcolors.OKCYAN}Se taggea y wallet drop {drop[0]} {add}: {config.crypto[user][wltdrop]}{config.bcolors.ENDC}")
            sleep(random.randrange(10, 15, 1))
            accion = 'Tag y Walletdrop'
        elif tg:
            api.update_status('@%s %s' % (tweet.author.screen_name, config.crew[user]), in_reply_to_status_id=tweet.id)
            print(f"{config.bcolors.OKCYAN}Se taggea{config.bcolors.ENDC}")
            sleep(random.randrange(10, 15, 1))
            accion = 'Tag'
        elif wltdrop is not False:
            api.update_status('@%s %s' % (tweet.author.screen_name, config.crypto[user][wltdrop]), in_reply_to_status_id=tweet.id)
            print(f"{config.bcolors.OKCYAN}Drop {drop[0]} {add}: {config.crypto[user][wltdrop]}{config.bcolors.ENDC}")
            sleep(random.randrange(10, 15, 1))
            accion = 'Walletdrop'
        else:
            print('Ni tag ni wallet')
            accion = 'No'
        return accion
    except Exception as e:
        print(str(e))

I think these are all the core functions of check

[Harmon758]

The IdIterator that Cursor returns when used with API.home_timeline stops when it receives a page with no results. This is most likely what's happening, since the default count for the endpoint is 20 and:

The value of count is best thought of as a limit to the number of tweets to return because suspended or deleted content is removed after the count has been applied.

https://developer.twitter.com/en/docs/twitter-api/v1/tweets/timelines/api-reference/get-statuses-home_timeline

This is a limitation of this Twitter API endpoint, as there's not another good way to determine when to stop paginating.

However, you can pass a higher count (e.g. 100 if that works for you, up to 200) to the endpoint while using Cursor with it and you'll be less likely to receive a premature empty page.

[Lueesla]

Okay, but, how does it receives a page with no results of my home timeline? It cant be reaching all the lifetime tweets of my timeline because that would be much more than 20 or 40 tweets

Anyway, lots of thanks Harmon, if its a limitation of the API then I just can find other way, I didnt know if it was my fault or not :)

[Harmon758]

The value of count is best thought of as a limit to the number of tweets to return because suspended or deleted content is removed after the count has been applied.

This means a page can have 0 results even when not having paginated through all the results yet. You can decrease the chance of this happening by passing a higher count to the endpoint.