Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Referer Header when fecthing image medium #305

Open
JingMatrix opened this issue Jul 21, 2023 · 3 comments
Open

Add Referer Header when fecthing image medium #305

JingMatrix opened this issue Jul 21, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@JingMatrix
Copy link

Describe the bug

Some images hosting services such as https://b-1254719278.cos.accelerate.myqcloud.com/wp-content/uploads will refuse to load images if no valid Referer header is present in the request.

To Reproduce
An examplar feed is: https://bedtimepoem.com/feed/atom
To see the issue, the command

curl -I 'https://b-1254719278.cos.accelerate.myqcloud.com/wp-content/uploads/2023/07/WechatIMG19-1024x1024.jpg'

returns 403.
But the command

curl -I 'https://b-1254719278.cos.accelerate.myqcloud.com/wp-content/uploads/2023/07/WechatIMG19-1024x1024.jpg' \
  -H 'Referer: https://bedtimepoem.com/archives/17372'

returns 200 correctly.

Expected behavior
rsstt should add the Referer header so that images can be fetched reliably.

Important log

Traceback (most recent call last):
  File "/home/jing/.local/python/lib/python3.11/site-packages/rsstt/web.py", line 457, in get_medium_info
    raise ValueError(f'status code is not 200, but {r.status}')
ValueError: status code is not 200, but 403
2023-07-21-16:21:12:DEBUG:RSStT.web - Medium fetch failed: https://b-1254719278.cos.accelerate.myqcloud.com/wp-content/uploads/2023/07/WechatIMG19-1024x1024.jpg
Traceback (most recent call last):
  File "/home/jing/.local/python/lib/python3.11/site-packages/asyncstdlib/_lrucache.py", line 407, in __call__
    result = self.__cache[key]
             ~~~~~~~~~~~~^^^^^
KeyError: 'https://b-1254719278.cos.accelerate.myqcloud.com/wp-content/uploads/2023/07/WechatIMG19-1024x1024.jpg'
@Rongronggg9
Copy link
Owner

Sounds reasonable. However, I am pretty occupied these days so PR is welcomed.

@Rongronggg9 Rongronggg9 added the enhancement New feature or request label Jul 21, 2023
@JingMatrix
Copy link
Author

Hi! Do you have related development documents? I didn't find them yet.

Most importantly, how do you set up a test environment locally?

@Rongronggg9
Copy link
Owner

Hmm, it seems that only RSStT fetching images with proper referer is not sufficient. RSStT relies on Telegram DC to finally fetch and send images. That RSStT itself succeeds in fetching images does not mean the success of Telegram DC, while it is the latter that is the key to sending images.
To fix the issue, https://github.com/Rongronggg9/rsstt-img-relay needs to accept custom HTTP headers first. Afterward, RSStT can benefit from it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants