looks like X/twitter(?) broke something again #983
Comments
broke again zedeus/nitter#983
|
Also, the syndication api for |
|
Yes, it is not working now. I hope the Nitter people fix this soon. |
|
Is there a online/CLI tool converting |
Not really. showReplies=false shows years-old content when not logged in. |
|
Down again... |
That's because in that specific example, those tweets were years ago. Look again at the like count, notice anything? |
We can just search for the first |
Indeed #!/usr/bin/python3
import requests
import re
import urllib
url = "https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk"
with urllib.request.urlopen(url) as response:
encoding = response.info().get_param('charset', 'utf8')
html = response.read().decode(encoding)
result = re.search('script id="__NEXT_DATA__" type="application\/json">([^>]*)<\/script>', html)[1]
print(result) |
Interesting, but this doesn't return RSS with 'item', 'pubDate' etc. tags. Maybe a script using https://github.com/lkiesow/python-feedgen would do the job? |
Not sure I understand ? It expose far more informations than needed and it does expose the date and all Here's an example for one tweet only : {
"type": "tweet",
"entry_id": "tweet-1519480761749016577",
"sort_index": "1691455400412446720",
"content": {
"tweet": {
"id": 0,
"location": "",
"conversation_id_str": "1519480761749016577",
"created_at": "Thu Apr 28 00:56:58 +0000 2022",
"display_text_range": [
0,
52
],
"entities": {
"user_mentions": [],
"urls": [],
"hashtags": [],
"symbols": [],
"media": []
},
"favorite_count": 4600599,
"favorited": false,
"full_text": "Next I’m buying Coca-Cola to put the cocaine back in",
"id_str": "1519480761749016577",
"lang": "en",
"permalink": "/elonmusk/status/1519480761749016577",
"possibly_sensitive": false,
"quote_count": 171975,
"reply_count": 187438,
"retweet_count": 649833,
"retweeted": false,
"text": "Next I’m buying Coca-Cola to put the cocaine back in",
"user": {
"blocking": false,
"created_at": "Tue Jun 02 20:12:29 +0000 2009",
"default_profile": false,
"default_profile_image": false,
"description": "Blades of Glory",
"entities": {
"description": {
"urls": []
},
"url": {}
},
"fast_followers_count": 0,
"favourites_count": 30569,
"follow_request_sent": false,
"followed_by": false,
"followers_count": 153112066,
"following": false,
"friends_count": 410,
"has_custom_timelines": false,
"highlightedLabel": {
"badge": {
"url": "https://pbs.twimg.com/profile_images/1683899100922511378/5lY42eHs_bigger.jpg"
},
"description": "X",
"userLabelType": "BusinessLabel",
"userLabelDisplayType": "Badge"
},
"id": 0,
"id_str": "44196397",
"is_translator": false,
"listed_count": 126597,
"location": "𝕏Ð",
"media_count": 1659,
"name": "Elon Musk",
"normal_followers_count": 153112066,
"notifications": false,
"profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1690621312",
"profile_image_url_https": "https://pbs.twimg.com/profile_images/1683325380441128960/yRsRRjGO_normal.jpg",
"protected": false,
"screen_name": "elonmusk",
"show_all_inline_media": false,
"statuses_count": 29441,
"time_zone": "",
"translator_type": "none",
"url": "",
"utc_offset": 0,
"verified": false,
"withheld_in_countries": [],
"withheld_scope": "",
"is_blue_verified": true
}
}
}
},EDIT : Maybe you meant a directly usable solution for an end user, and of course it's not, the snippet need to be adapted by a dev. |
Ok, thank you, I'll try this. |
calling the syndication URL without being logged in twitter doesn't retrieve the most recent tweets. If I call this url in postman, I retrieve 100 tweets from 10/19/2018 to 07/31/2023; no tweets from august... |
It retrieves the tweets with the highest like count from that user, which doesnt sound good if your goal is retrieving the most recent tweets, as there's no guarantee new tweets will make it to the top 100 tweets from that user. And even if they did, it might take a considerable amount of time |
|
I've noticed that for smaller accounts that have less than 100 tweets, that syndication URL does not load any tweets. |
No. That is the case for all big accounts. I am interested in the most recent Tweets and this approach will lead to nothing. |
im using this for my bot and it working fine with cookies and headers. |
Yes. And there is at least one tweet from August with more likes (>807K) than some older tweets which are included (e.g. <680K). |
|
is there any forecast for solving this problem? |
|
Looks like https://nitter.privacydev.net/ is working |
|
That one is a fork which uses account credentials. See #830 |
|
I am aware but couldn't nitter implement a system that aurora uses with lots of accounts that rotate per user? |
That's hard to maintain and simple for twitter to ban by just filtering "if number of accounts per IP > SOME_CONSTANT: ban all of them" |
User feeds not working on this |
|
I switched to the privacydevel fork, credentials in but its still 404ing the same endpoint upstream is having problems with |
|
Strange. privacydev (without credentials) works more or less for @ElonMusk, but not for other users like for instance @BarackObama. |
|
Twstalker is doing funny things: when someone quotes a tweet, the quoted tweet repeats what the quoter says. |
|
@animegrafmays Hey, how/where can I contact you to whitelist my IP for nitter.poast.org :) ? |
|
im not in the market of 'whitelisting' IPs. you can email me at graf @ poast.org and I can see why yours is blocked, but I'm not whitelisting people |
@animegrafmays Mine is blocked too, maybe because I use the RSS feeds for some accounts (~ 10). I set them to fetch new posts once per hour on my browsers, instead of every 30 minutes. Before I e-mail you, could you confirm that it could have been a cause of “ERR_CONNECTION_REFUSED”? |
|
this github issue is not an issue tracker for your connection issues to my nitter instance. please stop shitting up this github issue by triggering an email every single time you respond and instead use the contact methods i've presented in this thread already |
I have never had an issue with the RSS feeds lately, I get them fine with Power Automate and Element Feeds. Fully agree with the fact we are clogging up but it's not like I can move this to fediverse (the servers I am on block poa.st because of allowing some undesirable people that were on certain platforms that's all). |
The solution to all your problems is to host these services yourself. One of the common SBC or an old laptop - screen not required - is more than enough to run an instance of Pleroma (for your fediverse needs), Nitter, Libreddit, Invidious (Youtube proxy) and even Peertube if you go easy on the transcodes. That way you are neither a burden to others nor beholden to them. You may get blocked by the 'purple-hair crowd' on Mastodon but that is more or less unavoidable unless you want to become part of that crowd yourself. As long as you behave sensibly - and why wouldn't you? - you should be able to federate with the world and its dog, minus the aforementioned block-happy group. Of course all this assumes you either have a stable internet connection at home to which you can attach said SBC or laptop or thin client or what have you or can splurge a few |
|
For those of us with no landline, a need to self-host a service simply to access Twitter content is the same as Twitter not being usable.
|
Wow, amazing information. If I was able to, I would have done so already. I have no experience with hosting stuff and there's no chance of me running up a cable to connect a device. Using a VPS seems expensive too, a local company wants £3 for their lowest tier a month. @lukefromdc how is a landline related to this? |
If you only would use the machine for TwiXXer->Nitter, yes that would be overkill. The idea would be to use your own host for much more than that both to avoid the mentioned blocking problems as well as to host your own services. Host your own photos instead of giving them to Google, host your own music instead of being dependent on streaming services, host your own XMPP server instead of being dependent on Telegram/Whatsapp/Signal/Matrix or ( |
|
Landline relevence is much more bandwidth available. I have de facto (not admitted to) capped cellular bandwidth via tethering only, when I am on the road it would be out of range of any home server. Forget a static IP address of course. Also they don't like tethering but blocking sophisticated ways to tether brings lots of false positiives so they don't bother. If they do I get a new carrier and a new phone number. No stable IP address, no continuously on connection in any single location, not going to carry a server everywhere I go and we are not discussing the presumably lighter load of distributed hosting (e.g diaspora) here. I do not however use Google or Meta for any purpose as I oppose their business models. I publish videos on Mastodon and on archive.org (latter good for linking especially as they want copies of all online content anyway). Twitter used to be widely used in my community so I had to be able to keyword search it and pull up known useful timelines, now neither of those work on the Twitter webapp without an account. I refuse to make an account with any ad supported corporate social media site, but the importance of Twitter is plunging as people stop posting there. Unfortunately this is fueling a migration to Instagram which is even worse than Twitter but at least still has a partially working 3ed party backend. You must know at least the username whose content you want to see to use it as far as I can tell. On the plus side, the migration to Mastodon has greatly boosted the reach of my content, but some events I would cover now go unseen as they are organized in places I am now blind to. |
Ah that makes sense. Providers here generally don't require a landline but one provider still only does ADSL service and the other is HFC. |
|
so people stop generating multiple notifications via here and email for stuff that isn't related to nitter development, if it pertains to my nitter instance please email graf at poast dot org. do not post it here. want to be critical of me? email it. have a problem with what im doing? email it. leave this thread for dev. thanks |
Cheers for all the work that you do on Poast bud @animegrafmays Apologies in advance for the plethora of questions but I would appreciate some further tips before I start smashing my head against a wall trying to get a private instance working well enough for my needs. Given your experience running the most reliable Nitter instance as of late I would be very grateful for any advice around accounts usage and I'm sure I'm not the only one. Basically I need to scrape a few hundred accounts, their complete timelines including likes, complete list of followers and following. What would be the best way to go about it? Would it be more effective to whip up some quick scripts that query graphql rather than run a private nitter instance? I'm not as concerned about performance gains as I am with how nitter cycles through the accounts. I wonder if e.g. requesting the timelines in order with sane delays through some custom scripts would help as far as not getting the accounts blacklisted as much. I've also thought about queuing up all the tweets containing media and then spinning up a headless browser separately in order to request them directly from twitter, since you can seemingly still do this much unauthenticated. Or is this overkill? You've mentioned aged accounts faring better but most people, myself included, don't have access to any. Although the scale of private instances is infinitely smaller than the one you run, I'm still wondering what measures I could take to optimize fresh accounts usage. Thank you in advance for any insight which will result in less hair pulling on my part. |
i run private instances for this so they don't have impact on the public instance, you can reach out to me at graf [at] poast.org if you are interested.
for what you are asking, if you are doing a one-time thing, running a quick nitter with some even new-ish accounts would be fine. the limits for accounts older than I think 2018 are doubled whereask those prior to 2012 seem to be tripled or maybe quadrupled. i can't find documentation on this but in practice we run 200 accounts older than 2012 and very rarely run into limited accounts beyond maybe a dozen out of that in a day. older account limits are much, much more lax and definitely the way to go if you can get your hands on them
if you are using RSS it's limited to the cache time of the nitter instance. nitter.poast.org is on 30 minute refresh for rss, staggering requests on a private instance (i.e. request 5 accounts now, 5 more in 5 minutes, etc) would help spread out user account exhaustion but in my experience as I said prior you can more or less ignore this with aged accounts. banned accounts have limits almost identical to those created in the last 12 months so if you can get your hands on a bunch of tokens from those and you're just serving yourself/your own interests its more than adequate. it's a lot easier to find banned/suspended accounts from people than current, active ones
you mean fetching a list of attachments on the tweet and bulk downloading? seems a bit overkill. if you are running a private nitter instance you should be fine without needing to go to this extreme
we happen to have a relatively large userbase (32k users at time of writing) who had donated a bunch of accounts and I have some that I had left over. I also have some surplus I could give you tokens for if you're using a private instance for yourself to get you started. we have 22-25k requests/s avg (just recently had to move it to a ryzen 7 server because the core clock on individual threads wasn't enough even running multiple nitter processes)
to be honest, as long as you aren't listing yourself on the wiki or on the status page you likely wont run into issues. new accounts have about ~200 profile queries each that seem to reset between 8-12 hours, so a private instance with maybe 5 or 6 accounts would do you with staggered scraping for quite some time i dont want to clog up this issue or notify everybody so if you'd like to discuss it further you can email me at the address above |
|
Why are you even engaging in a convo with someone that "needs to scrape a few hundred accounts" and then offering to help with his endeavour? I must be taking crazy pills without knowing. |
|
Twitter's fault for not providing an API for this. There's nothing wrong with scraping. |
I'm not going to take your bait and get bogged down in a pointless debate re the ethics of scraping and how it all might pertain to Twitter's newest policies. From a pragmatic standpoint, several Nitter instance maintainers have mentioned in this very thread that scraping is a big enough problem as far as keeping Nitter usable for regular users. Since I prefer to use Nitter myself for casual lurking rather than having to deal with Twitter's interface and algo, the last thing I wanted to do was contribute to the problem. I believed addressing this out in the open would go a long way towards helping those with this specific need get their own private instances up and running, rather than ruin a good thing for everyone. Taking my own use case as an example, since speed is not a concern it would've certainly been more convenient to scrape graf's instance for weeks rather than have to deal with Twitter socks, but it's a really shitty thing to do. As for why someone might be interested in scraping Twitter on a small scale - several hundred accounts is nothing really despite your scaremongering - there are legitimate reasons. From Twitter's shitty built-in search to archival purposes; some communities sharing precious tidbits on Twitter are already more or less shutting down in protest because Elon man bad. |
i'll break it down for you -- i'd rather somebody scrape an instance i run that nobody else is trying to use than use my main instance, that literally everybody is linking to. cool with you? no? i dont give a shit |
|
oh hey, emails for that Nitter issue are back! I wonder if they ever fixed the probl—
|
Scraping is why we cant have nice things such as Nitter in the public as it used to work before "some" started to scrap Twitter through Nitter and put Nitter to a halt. |
|
nitter.poast.org sometimes doesn't show search results or individual Tweets. nitter.privacydev.net shows the rate limit error on all searches and account pages even with a full bar of green at the Nitter status. It'd be great if people started working on a solution as opposed to dooming every moment about the death of Nitter 😐 |
|
the reason searches are "sometimes" not shown is due to the amount of accounts limited. there's currently 200 accounts on that instance and because it's serving traffic to almost 200 thousand unique visitors daily more than half of them are rate limited. with that amount of traffic having more than half the accounts rate limited will result in not being able to expand tweets, searches not functioning etc. it's a literal full time job maintaining it and preventing scraping but I am doing what I can including procuring an additional 100 to add to it this week. sorry I can't be good enough for you, @nkfm200 😭😭😭 |
|
I continue to be amazed that you've got this running so long after development ceased |
|
@animegrafmays That I've heard of, the small number of accounts serving a huge number of visitors every day causing rate limits on accounts. Hopefully that extra 100 accounts is there at the time when it's the end of the week. This may be off-topic but I'm just strict about stuff because this is a world where the people living in it refuse to do something unless it fits the rules of the culture |
and others
The text was updated successfully, but these errors were encountered: