[multiple sites] Error solving the challenge. Timeout after X seconds - challenge loop #1036
Comments
ilike2burnthing
added
help wanted
|
Debug logs and headless=false both confirm that the challenge is found, box ticked, page refreshed, but the challenge just returns. Tested on both Windows and Docker. This was the same behaviour seen with yggtorrent, which was resolved by adding the ENV If anyone has any ideas, or it's working for anyone, let me know. |
|
I have the same issue on multiple other sites, doesn't look like a site specific thing |
|
3.3.9-3.3.13 |
|
I have the same issue on another totally different site. I had to remove the call from the script because there was no challenge. The script went clean for now. I'll try later an older version. I'm running it on a VM in Proxmox alongside other docker apps like sonarr, radarr, etc |
This comment was marked as duplicate.
This comment was marked as duplicate.
ilike2burnthing
changed the title
|
I think the issue might be related to using sessions. I previously was using them and in general it worked, but for some sites it would fail after a few requests in a timeout. Switching to a standard cache of cookies and returning them in the get request solved it for me. This probably is very situational and does add more processing time I would imagine since I am spinning up more headless instances, but it worked for me. |
How do you do this? Sorry if that is a dumb question. |
Cache the cookies from FlareSolverr and then send them back in your new requests. |
Hi @DHuckaby, would you mind sharing an example on how to do this? |
|
Hi @ilike2burnthing, what I just found out is that it can take a few seconds to load the "verify I am a human box", even when using a regular browser. I guess it's checking the IP address validity, before showing the challenge. In my case, it only finds the challenge very occasionally. Therefore, is it a possibility to add a (configurable) timeout that awaits for the challenge to appear on the page? Or maybe let it check multiple times if the button is displayed on the page (referring to this line in the code ) ? |
|
FlareSolverr already does this. Enable debug logging and you'll see it cycling through the check multiple times. |
# Copy of existing Python example on README
import requests
url = "http://localhost:8191/v1"
headers = {"Content-Type": "application/json"}
data = {
"cmd": "request.get",
"url": "http://www.google.com/",
"maxTimeout": 60000
}
response = requests.post(url, headers=headers, json=data)
print(response.text)
# Extract cookies from solution response if successful
cookies = []
json_response = response.json()
if json_response["status"] == "ok":
cookies = json_response["solution"]["cookies"]
# New request with previous request's cookies
response2 = requests.post(url, headers=headers, json=data, cookies=cookies)
print(response2.text) |
|
That doesn't seem to be working for me - At least for fanfiction.net, when I'm just trying to do the initial request to get a cookie. |
agree, I have same problem |
|
I'm not sure if this issue is related, but I have face similar issues somewhere else... On the android app for reading manga, Tachiyomi (currently stopped development but there's many forks including Mihon, SY, J2K etc...), I often face cloudflare issue for the source i'm reading. Will have to open a build-in browser then manually solve the cloudflare. Some sources can be solved manually with the build-in browser, however, sources like Happymh has very strict cloudflare, and we have to change the user-agent in the app so that cloudflare would not get the challenge loop. Perhaps playing with different user-agent could help? Currently I've set to:
Also, in another development Tachidesk, due to it's nature of running as a server, it does not have a "native browser", so we could not solve the cloudflare manually, and recently they've added Flaresolverr. But folks over there said Flaresolverr doesn't have a function to change it's user-agent (I'm not sure...), so the challenge loop also occurs. I wish I could code (only understand very very basic coding) to help. And I hope this could help the communities if it does solve the issue everyone is facing here. |
|
user-agent header isn't supported, hasn't been since v2, over 2yrs ago. |
|
So could the user-agent be the issue for the cloudflare challenge loop? |
|
Possibly, but I can't check. |
I confirm, I added the user-agent and it worked again Line 132 in df06d13 Just add after line before: |
|
Sweet... So shall we add this feature back? and also allow a var in the config to change the user-agent? |
|
While the ability to use an ENV to achieve this could be added, previously it was part of both FlareSolverr and FlareSolverrSharp, and could be used by indexers which required cookie and UA login. I'll have a look later, but I doubt I'll be able to recreate this. PRs welcome. |
|
I'm current using the unraid version. Is it possible to just add a ENV VAR and set the value to it? What should the VAR be? |
|
Edited comment above to clarify. No, an ENV cannot currently be used. |
|
Started working for me this morning with no changes, must be magic :) Latest release. |
|
Still failing on ffnet though
…On Wed, May 1, 2024, 8:02 PM Paul Morris ***@***.***> wrote:
Started working for me this morning with no changes, must be magic :)
Latest release.
—
Reply to this email directly, view it on GitHub
<#1036 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAMW5G7DE6G3WAT3MUOWWODZAF67ZAVCNFSM6AAAAABBXLAM52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBZGMYTEMBSHA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
i don't know the code so i don't know if its possible but could a popup full browser window for you to manually respond in response to failure let you generate working cookies as a workaround? |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
|
This docker image work on my end, on k8s setup: Setup an LANG=fr-FR worked nice for yggtorrent :) |
|
So... take this with a grain of salt, cause it's not exactly the most scientific test I've ever done... I was annoyed to the point where I made a couple of tweaks to my docker instance:
Getting this far, the browser bot checkers only really flag the timezone discrepancy, my real browser seems to have more sus results. At which point I made an odd discovery... if I requested the page it span up the Chrome instance, navigate to the page and promptly went into the auth loop (where it clicked the cf button and the page that loads is back to the cf page instead of the site)... if, in the same window, I quickly open a new tab, manually navigate to the same url, and click the checkbox... yeah, that succeeds first time, which the other tab then picking up the cookies and the rest of the solverr sequence completing normally. Since this should, logically at least, mean the environment or browser itself isn't the reason for detection and that leaves me suspecting one or more of three potential reasons:
One other thing I notice, though it might be a Herring of the Red variety, is that while trying to coax the loop into giving me the protected page I was getting the notification for blocked Third-Party Cookies around when the page navigated. I didn't dig into finding which third party though, so it could be CF, the site behind, or one of the many external I can't do a proper dig around right now, not got free slots on my todo list and have resorted to just baby sitting the browser process via vnc for now. |
This comment was marked as duplicate.
This comment was marked as duplicate.
|
Hello, |
This comment was marked as spam.
This comment was marked as spam.
|
More work needs to be done with the current PR and/or the fork by 21hsmw. Read my comments on the PR for more info. |
|
Half work, still got a an error during the workday. i will plug the output the log on a file tomorrow and try to grab the errors. increased my timeout to 180 from 60 too |
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
Great how do i make the change ? can you make a fix ? |
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
|
Due to 'same here' and 'when will this be fixed' spam, this issue is now locked. Read back on previous comments (you should be doing that anyway) if you want to know more and about the current PR. New PRs or constructive contributions to current PRs are always welcomed. Opening new issues to try to circumvent this will result in a ban. Commenting on other issues or PRs to try to circumvent this will result in a ban. Opening new issues has now been restricted to what GH deems 'existing users', due to the number of new accounts (particularly those using YGGtorrent) refusing to read old issues before posting, despite ticking a box saying they definitely did so. |
and others
Have you checked our README?
Have you followed our Troubleshooting?
Is there already an issue for your problem?
Have you checked the discussions?
Environment
Description
Using FanFicFare to scrape from fanfiction.net. Nothing's changed with my config, but it stopped working this week.
Logged Error Messages
Screenshots
No response
The text was updated successfully, but these errors were encountered: