[refactor] fix Profile object #167

yimingStar · 2020-08-19T08:10:20Z

Uable to get the original page (400 error code), get mobile twitter page indead.

Change the User-Agent
Fix the selector queries (Missing getting birthday, favorite and is_verified)
UnboundLocalError: local variable 'html' referenced before assignment #166

1. is_private 2. birthday 3. favorites

yeachan153 · 2020-09-09T21:22:02Z

Hey, just passing and I don't really know the code enough to do a PR. Just to let you know I tried your branch and it did not work for me, although the issue is not related with the user agent I think.

yeachan153 · 2020-09-09T21:25:21Z

twitter_scraper/modules/profile.py

            self.location = None

+        # TODO unfixed
        self.birthday = html.find(".ProfileHeaderCard-birthdateText")[0].text


Not too familiar with the code (and appreciate it's not the issue you are fixing), but I would assume to make the code work it would be better to put this in a try except IndexError block?

Hi, thank you for the response!
The error is an error due to the empty list, I have added an exception catch for all elements, please try again.
I did not use specific IndexError is because the original code did not specify Errors, so I think it will be better to use the same style.

Once again thank you for disclosing the problem!

profile = Profile(f"{user_id}") print(profile.to_dict())

You could try using the code by the fixed version above, I use "BarackObama" as user_id to gets the output below:

output

{'name': 'Barack Obama', 'username': 'BarackObama', 'birthday': None, 'biography': 'Dad, husband, President, citizen.', 'location': 'Washington, DC', 'website': 'obama.org', 'profile_photo': 'https://pbs.twimg.com/profile_images/822547732376207360/5g0FC8XX_normal.jpg', 'banner_photo': None, 'likes_count': None, 'tweets_count': 15921, 'followers_count': 122418592, 'following_count': 601466, 'is_verified': True, 'is_private': False, 'user_id': None}

New version is working! Thanks a lot!

@yeachan153 Thank you too ~ could you please confirm the review request?

@yeachan153 Thank you too ~ could you please confirm the review request?

Seems valid. Could you please merge it? It's been quite a while, everything is better than the current version that is not working with profiles at all.

yeachan153

The last time I checked, the code is working! That's really nice!

I am still a bit concerned about the bare try except blocks; while keeping in the style of the code can give unexpected results in the future. I.e. if the HTML changes, then user attributes can be assigned None without raising an explicit error and could lead to confusing situations and raise cases where mistakes are caught much later because we assume that everything is working as intended.

I am not sure what a perfect solution would be. Maybe errors should be logged even if the scraper continues, so we are aware what went wrong even if the final result is outputted? Just a suggestion.

But nice work on getting everything to work :-)

aldnav

Thanks for the fix @yimingStar

Seems that soon enough we'd be needing to delegate these properties to a general function because it follows a pattern (find from selector, get first, get text, return value else default to None), and any deviation from that would have to become it's own function.
Generally it's simple and works as it is right now.

I left a suggestion. Let me know what you think.

aldnav · 2020-09-25T01:38:31Z

twitter_scraper/modules/profile.py

+        stats_table = None
+        stats = None
+        try:
+            stats_table = html.find('table.profile-stats')[0]
+            stats = stats_table.find('td div.statnum')
+            if not stats:
+                self.tweets_count = None
+                self.following_count = None
+                self.followers_count = None
+        except:
+            self.tweets_count = None
+            self.following_count = None
+            self.followers_count = None


Suggested change

stats_table = None

stats = None

try:

stats_table = html.find('table.profile-stats')[0]

stats = stats_table.find('td div.statnum')

if not stats:

self.tweets_count = None

self.following_count = None

self.followers_count = None

except:

self.tweets_count = None

self.following_count = None

self.followers_count = None

try:

stats_table = html.find('table.profile-stats')[0]

stats = stats_table.find('td div.statnum')

except:

stats_table = None

stats = None

How about this?

@aldnav Thank you for your suggestion. Your suggestion is definitely better!
I just make an adjustment in this refactor.
The fixed is in this comment f13ec7f

stats - tweets_count - following_count - followers_count

yimingStar · 2020-09-29T07:00:49Z

@yeachan153 Genuinely agree with your suggestion for the purpose of letting the user notice the missing web elements.
So I added the exception function into this Profile class, which will print out the missing attributes.

The fixed is in this commit c7bbf55

aldnav

The fix works. Left a minor comment. Overall LGTM. Thanks @yimingStar 👍

aldnav · 2020-10-01T23:50:52Z

twitter_scraper/modules/profile.py

@@ -39,7 +40,7 @@ def __init__(self, username):
        page = session.get(f"https://twitter.com/{username}", headers=headers)
        self.username = username
        self.__parse_profile(page)
-
+        


Nitpicking, extra white spaces.

andreasvc · 2020-10-17T16:54:59Z

twitter_scraper/modules/profile.py

@@ -49,77 +50,128 @@ def __parse_profile(self, page):
            )
        except ParserError:
            pass


You shouldn't ignore a fatal error.

that was in the original package, so it's not related to this PR. Still, it should be ignored

HJoentgen · 2021-03-16T13:29:31Z

Is this solution still working?
When I use the new profile.py, I still get the same "local variable 'html' referenced before assignment" error, because of a 400 response.

So this:

from twitter_scraper import Profile
profile = Profile(f"BarackObama")
print(profile.to_dict())

leads to:

Unable to get is_private in html, exception - local variable 'html' referenced before assignment
...

EricChangGo added 2 commits August 18, 2020 18:18

[refactor] fix Profile object

b679c68

[comment] adding TODO

9fa7443

1. is_private 2. birthday 3. favorites

yimingStar mentioned this pull request Sep 2, 2020

UnboundLocalError: local variable 'html' referenced before assignment #166

Open

yeachan153 reviewed Sep 9, 2020

View reviewed changes

yimingStar added 2 commits September 10, 2020 11:21

[refactor] add exception for all column

5bb60d3

adding TODO comments

cb8244e

yeachan153 reviewed Sep 24, 2020

View reviewed changes

aldnav reviewed Sep 25, 2020

View reviewed changes

EricChangGo added 2 commits September 29, 2020 14:50

[refactor] adding exceptions for all attributes

c7bbf55

[refactor] rearrange logic of getting few attributes

f13ec7f

stats - tweets_count - following_count - followers_count

aldnav approved these changes Oct 1, 2020

View reviewed changes

andreasvc reviewed Oct 17, 2020

View reviewed changes

HJoentgen mentioned this pull request Mar 16, 2021

Not get Profile Info #180

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[refactor] fix Profile object #167

[refactor] fix Profile object #167

yimingStar commented Aug 19, 2020

yeachan153 commented Sep 9, 2020

yeachan153 Sep 9, 2020

yimingStar Sep 10, 2020

yimingStar Sep 10, 2020

yeachan153 Sep 10, 2020

yimingStar Sep 24, 2020 •

edited

DataGreed Nov 9, 2020

yeachan153 left a comment •

edited

aldnav left a comment

aldnav Sep 25, 2020

yimingStar Sep 29, 2020 •

edited

yimingStar commented Sep 29, 2020 •

edited

aldnav left a comment

aldnav Oct 1, 2020

andreasvc Oct 17, 2020

DataGreed Nov 9, 2020

HJoentgen commented Mar 16, 2021 •

edited

[refactor] fix Profile object #167

Are you sure you want to change the base?

[refactor] fix Profile object #167

Conversation

yimingStar commented Aug 19, 2020

yeachan153 commented Sep 9, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yimingStar Sep 24, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yeachan153 left a comment • edited

Choose a reason for hiding this comment

aldnav left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yimingStar Sep 29, 2020 • edited

Choose a reason for hiding this comment

yimingStar commented Sep 29, 2020 • edited

aldnav left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HJoentgen commented Mar 16, 2021 • edited

yimingStar Sep 24, 2020 •

edited

yeachan153 left a comment •

edited

yimingStar Sep 29, 2020 •

edited

yimingStar commented Sep 29, 2020 •

edited

HJoentgen commented Mar 16, 2021 •

edited