Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functionality to fetch guest accounts on demand #1116

Open
wants to merge 9 commits into
base: guest_accounts
Choose a base branch
from

Conversation

drpepper66
Copy link

Add a basic way to automatically fetch guest accounts from an external URL if the internal stash is empty.

for example:

guestAccountsUrl = "https://twitterminator.x86-64-unknown-linux-gnu.zip/download"
guestAccountsKey = "<random string>"

If those settings are configured, guest_accounts.json does not need to be present.

this serves the hashed Key under /.well-known/nitter-auth-sha256, while twitterminator expects it under /.twitterminator.txt. If this approach is accepted I'll update twitterminator to look for the new path too.

Comment on lines 36 to 40
# Instead of guest_accounts.json, fetch guest accounts from an external URL.
# Nitter will re-fetch accounts if it runs out of valid ones.
guestAccountsUrl = "" # https://example.com/download
guestAccountsHost = "" # defaults to nitter's hostname
guestAccountsKey = "" # a random string that will be used as a secret to authenticate against the URL
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's put this into a new section, and also remove the tokenCount + comment above since that's no longer needed. This section can be used for other variables in the future like whether to try fetching accounts by itself, if so which proxy/proxies to use, jsonl file location, and some other tweaks to the auth handling.

Suggested change
# Instead of guest_accounts.json, fetch guest accounts from an external URL.
# Nitter will re-fetch accounts if it runs out of valid ones.
guestAccountsUrl = "" # https://example.com/download
guestAccountsHost = "" # defaults to nitter's hostname
guestAccountsKey = "" # a random string that will be used as a secret to authenticate against the URL
# Instead of guest_accounts.jsonl, fetch guest accounts from an external URL.
# Nitter will re-fetch accounts if it runs out of valid ones.
[GuestAccounts]
usePool = false # enable fetching accounts from external pool
poolUrl = "" # https://example.com/download
poolAuth = "" # random secret string for authentication
poolId = "" # defaults to nitter's hostname
poolAmount = 10000 # amount of accounts to request from pool

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

poolAmount = 10000 # amount of accounts to request from pool

I did not implement this. twitterminator does not allow you to negotiate the count, not in public api...

src/auth.nim Outdated Show resolved Hide resolved
src/auth.nim Outdated Show resolved Hide resolved
src/auth.nim Outdated Show resolved Hide resolved
src/routes/auth.nim Outdated Show resolved Hide resolved
src/auth.nim Outdated Show resolved Hide resolved
src/auth.nim Show resolved Hide resolved
src/auth.nim Outdated Show resolved Hide resolved
src/auth.nim Outdated Show resolved Hide resolved
src/auth.nim Outdated Show resolved Hide resolved
src/auth.nim Outdated Show resolved Hide resolved
@drpepper66
Copy link
Author

some features that are not implemented:

  • multi-node setups with load balancers. I think this could be supported by adding a sharding parameter to nitter config (or envvar): two variables ("shard ID", "shard count"), and nitter would only use the guest accounts where guest_account_id % shard_count = shard_id -- so you could even use the same guest_accounts.jsonl over multiple nodes

  • storing the fetched guest accounts somewhere -- right now nitter will not start up if the accounts service is down, and restarts of nitter will refetch accounts. I was thinking one could just store the accounts in redis to improve reliability and startup speed a bit

src/auth.nim Outdated Show resolved Hide resolved
@zedeus zedeus mentioned this pull request Nov 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants