No flats coming through on Immoscout #577
Comments
|
Same thing here. Sorry don't have any solution yet. I also get this error: I guess this is the main issue and then the page just times out or something and get's detected as bot. |
|
I think I have the same problem - Immoscout stopped working 2 days ago. However I don't think it's got to do with 2captcha. However, I can't really wrap my head around what the error message I'm getting tells us. I'm using docker-compose on a Linux server with 2captcha enabled. This is the log from after starting the container: |
|
I am not running with flathunter but my own crawler for Immoscout. I noticed that they seem to have switched from Geetest to AWS WAF. I haven't had time to look into yet, but 2captcha seems to support that kind of challenge. Just needs to be implemented. |
|
When I am running the same configuration (I am running seleniumbase instead of uc, but shouldn't matter too much) from my local machine via WSL it works and asks for a geetest captcha but on my VPS running directly on Linux it will return this pagesource:
As you said it will give us a awswaf captcha instead of the geetest. Could it be that for 'bad ips' there is another layer on captcha protection? Does something like this exist. I am not really deeply into it. |
|
I'm still seeing geetest:
According to the 2captcha docs, we need to captcha these details in order to solve a waf captcha: I can see |
|
There seem to be different versions of the Captcha challenge. One with key, context, iv, and then there is another one with only the challengeScript. I checked 2captcha docs and did not spot an api call version for the latter one. But I also checked only briefly I will attempt an implementation early next week and share my results here. But maybe someone else is keen in doing so already :-) |
|
To reproduce it I just need to use my VPS. I think the reason might be that it got flagged as a bad actor due to the amount of requests. It still works locally with geetest. But there only with the AWS captcha. The problem? (Dunno, have no experience there) is that the captcha only appears after the corresponding javascript is run (when I click inspect element). Then you get the needed information: One would need to map the type of tests to have Capsolver solve the captcha: https://docs.capsolver.com/guide/recognition/AwsWafClassification.html Ah and the challenge rotates pretty fast. Hopefully that won't be another issue (actually like 60 seconds) |
|
You only really need to challengeScript, which is available when the challenge is AWS WAF. I've created a basic example to dissect the challenge, submit it to Capsolver, and retrieve the result in form of a cookie. I'll just post the example here in case anyone would like to pick it up. from seleniumbase import Driver
import re
import requests
from time import sleep
from selenium_stealth import stealth
from selenium import webdriver
CAPSOLVER_API_KEY = "XXX"
CAPSOLVER_API_ENDPOINT = "https://api.capsolver.com/createTask"
url = "https://www.immobilienscout24.de/Suche/de/wohnung-mieten?sorting=2&pagenumber=1"
client = requests.Session()
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("--headless")
driver = Driver(uc=True, headless=True, agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36", uc_cdp_events = True)
driver.set_page_load_timeout(20)
driver.execute_cdp_cmd('Network.enable', {})
stealth(driver,
languages=["en-US", "en"],
vendor="Google Inc.",
platform="Win32",
webgl_vendor="Intel Inc.",
renderer="Intel Iris OpenGL Engine",
fix_hairline=True,
)
def delete_cookies():
for cookie in driver.get_cookies():
driver.delete_cookie(cookie['name'])
def get_aws_cookie():
for cookie in driver.get_cookies():
if(cookie['name'] == 'aws-waf-token'):
return cookie
def resolve_aws(iter=0):
script_content = driver.page_source
key_match = re.search(r'"key":"([^"]+)"', script_content)
iv_match = re.search(r'"iv":"([^"]+)"', script_content)
context_match = re.search(r'"context":"([^"]+)"', script_content)
jschallange_match = re.search(r'<script src="(.*?challenge.js.*?)".*?></script>', script_content)
key = None
iv = None
context = None
jschallange = None
if key_match and iv_match and context_match:
key = key_match.group(1)
iv = iv_match.group(1)
context = context_match.group(1)
jschallange = jschallange_match.group(1)
data = {
"clientKey": CAPSOLVER_API_KEY,
"task": {
"type": "AntiAwsWafTaskProxyLess",
"websiteURL": driver.current_url,
"awsKey": key,
"awsIv": iv,
"awsContext": context,
"awsChallengeJS": jschallange
}
}
else:
jschallange = jschallange_match.group(1)
data = {
"clientKey": CAPSOLVER_API_KEY,
"task": {
"type": "AntiAwsWafTaskProxyLess",
"websiteURL": driver.current_url,
"awsChallengeJS": jschallange
}
}
try:
task_id_response = client.post(CAPSOLVER_API_ENDPOINT, json=data)
task_id = task_id_response.json()['taskId']
try_cnt=0
while True:
cookie_response = client.post("https://api.capsolver.com/getTaskResult", json={"clientKey": CAPSOLVER_API_KEY, "taskId": task_id}).json()
sleep(5)
if cookie_response["status"] == "ready":
cookie = cookie_response["solution"]["cookie"]
# Replace the old cookie with the newly obtained
old_cookie = get_aws_cookie()
new_cookie = old_cookie
new_cookie['value'] = cookie
delete_cookies()
driver.add_cookie(new_cookie)
driver.uc_open_with_reconnect(driver.current_url, reconnect_time=3)
return True
elif cookie_response["status"] == "failed":
return False
else:
try_cnt+=1
if(try_cnt>5):
return False
continue
except Exception as e:
print(e)
# First delete all cookies, fetch IS24 page and solve AWS if presented
delete_cookies()
driver.uc_open(url)
if re.search("awswaf", driver.page_source):
resolve_aws(0) |
If this is really the case and 'bad ips' get flagged and shown the AWS captcha, wouldn't one temporary solution be to randomise the crawling interval by a few secs/mins? I don't have any other idea as to how they would sense it's a 'bad ip', as ~10mins seems like a totally reasonable refreshing time for an actual human. It might just be about the regularity? |
|
@jukoson Thanks so much for the sample code! It shouldn't be too hard to integrate that into the crawlers that we have. I don't know if I'll get around to that this week, but if someone else wants to give it a go that would be very welcome! |
|
You can solve using the documentation: |
The code above already resolves the captcha - however it does not yet bring you to the actual page you wanted to land at. I've progressed a little in that I can now solve the challenge and land on the is24 main website. From there, I accept cookies, type a search and then click the Search button. However another challenge pops up, after which I'm being redirected to the main page again. UPDATE from seleniumbase import Driver
import re
import requests
from time import sleep
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from random import randint
url = "https://www.immobilienscout24.de/Suche/de/wohnung-mieten?sorting=2&pagenumber=1"
CAPSOLVER_API_ENDPOINT = "https://api.capsolver.com/createTask"
CAPSOLVER_API_KEY = "XXX"
client = requests.Session()
DEFAULT_IS24_URL='https://www.immobilienscout24.de/'
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("--headless")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
userAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
driver = Driver(uc=True, headless2=True, agent=userAgent, uc_cdp_events = True)
driver.set_page_load_timeout(20)
driver.execute_cdp_cmd('Network.setBlockedURLs',
{"urls": ["https://api.geetest.com/get.*"]})
driver.execute_cdp_cmd('Network.enable', {})
def delete_cookies():
for cookie in driver.get_cookies():
driver.delete_cookie(cookie['name'])
def get_aws_cookie():
for cookie in driver.get_cookies():
if(cookie['name'] == 'aws-waf-token'):
return cookie
def resolve_aws(iter=0):
print(f"--------------------- ITER {iter}")
script_content = driver.page_source
key_match = re.search(r'"key":"([^"]+)"', script_content)
iv_match = re.search(r'"iv":"([^"]+)"', script_content)
context_match = re.search(r'"context":"([^"]+)"', script_content)
jschallange_match = re.search(r'<script src="(.*?challenge.js.*?)".*?></script>', script_content)
key = None
iv = None
context = None
jschallange = None
if key_match and iv_match and context_match:
key = key_match.group(1)
iv = iv_match.group(1)
context = context_match.group(1)
jschallange = jschallange_match.group(1)
data = {
"clientKey": CAPSOLVER_API_KEY,
"task": {
"type": "AntiAwsWafTaskProxyLess",
"websiteURL": driver.current_url,
"awsKey": key,
"awsIv": iv,
"awsContext": context,
"awsChallengeJS": jschallange
}
}
else:
jschallange = jschallange_match.group(1)
data = {
"clientKey": CAPSOLVER_API_KEY,
"task": {
"type": "AntiAwsWafTaskProxyLess",
"websiteURL": driver.current_url,
"awsChallengeJS": jschallange
}
}
try:
task_id_response = client.post(CAPSOLVER_API_ENDPOINT, json=data)
task_id = task_id_response.json()['taskId']
try_cnt=0
while True:
cookie_response = client.post("https://api.capsolver.com/getTaskResult", json={"clientKey": CAPSOLVER_API_KEY, "taskId": task_id}).json()
sleep(3)
if cookie_response["status"] == "ready":
# Get the cookie (AWS WAF token) from the CAPSOLVER response
cookie = cookie_response["solution"]["cookie"]
old_cookie = get_aws_cookie()
driver.delete_cookie('aws-waf-token')
new_cookie = old_cookie
new_cookie['value'] = cookie
driver.add_cookie(new_cookie)
captcha_container = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "captcha-container")))
interactive_element = WebDriverWait(captcha_container, 10).until(EC.element_to_be_clickable((By.XPATH, "//button | //input | //a")))
interactive_element.click(). <---------- There is a mistake here, it's not clicking the submit button!
print("Capsolver Solved")
if(driver.current_url == DEFAULT_IS24_URL):
print(f"Landed on DEFAULT_IS24_URL {DEFAULT_IS24_URL}")
driver.sleep(randint(1,3))
try:
shadow_root_ele = driver.find_element(By.CSS_SELECTOR, "#usercentrics-root").shadow_root
shadow_root_ele.find_element(By.CSS_SELECTOR, "button[data-testid='uc-accept-all-button']").click()
except Exception as e:
print(f"No cookie banner? e: {e}")
print("past the cookie banner!")
driver.sleep(randint(1,3))
if (driver.execute_script("return document.querySelectorAll('#oss-location')[1].value;") != 'Berlin'):
driver.type("(//input[@id='oss-location'])[2]", "Berlin")
driver.sleep(randint(1,3))
try:
# No idea why single click doesnt work
print("Clicking search")
driver.click('button.oss-main-criterion.oss-button.button-primary.one-whole.vertical-center-container')
driver.click('button.oss-main-criterion.oss-button.button-primary.one-whole.vertical-center-container')
except Exception as e:
print(f"Couldnt click? e: {e}")
driver.sleep(randint(1,3))
return True
elif cookie_response["status"] == "failed":
print("capsolver failed")
return False
else:
print(f"capsolver not ready yet.... Status: {cookie_response["status"]}")
try_cnt+=1
if(try_cnt>5):
print("capsolver did not process in time for the loop")
return False
continue
except Exception as e:
print(f"Resolve AWS WAF failed with {e}")
delete_cookies()
driver.uc_open(url)
if re.search("awswaf", driver.page_source):
print("AWS WAF Challenge")
resolve_aws(0)
if re.search("awswaf", driver.page_source):
resolve_aws(1)
if re.search("awswaf", driver.page_source):
resolve_aws(2) |
|
is it by design that you
Not criticizing or trying to gotcha you are or anything just wanting to unterstand the code. I am not sure the capsolver actually worked. Have you tasted it anywhere else? Instead of redirecting to main I try to use the 'submit' button after switching the cookies. But this didn't work. It said wrong answer. I replaced captcha_container and interactive_element with: I am not sure how the solver is supposed to work, but changing the cookie and then clicking ok is apparently not it |
|
@fmmix It was never intended to return to the webpage - I gave ChatGpt parts of the website and asked it to click that button for me to be honest. I didn't question it since I also did not notice any other buttons on the page. I could not explain why resolving the challenge would bring me to the main pafge, so I thought it's just some extra layer or sanity check that I'm not a bot. Now it all makes sense however. |
|
I don't have any experience with stuff like this myself, digging through website elements etc. That line here will tell the driver to click on the top left symbol which you already guessed. ` ` The issue with the page is that the submit/bestätigen button is behind a #shadow-root which will hide the elements from your normal find elements functions. But it looks like the idea is to not have to press any buttons by replacing the cookie it should just solve the problem by refreshing the page. I tried it using the 202 example from the blog: I played a bit more and almost got it to work. Not a full solution just the page_source part. I modified the while into a longer wait. And I run everything headed since I want to see what it does :-D, in headless2=True under with SB (...) it works just the same with getting the page_source. Maybe someone else can solve the missing puzzle? Or find another captcha solver for aws.Update: Another thing I found out is that the length of my manual cookie values are 326 and from the captcha solver only 262. Looks like the solver is simply not working correctly |
|
Thanks @fmmix , really good work. I feel this is getting somewhat closer to a solution. I just checked here as well, and the cookie length that I receive from capsolver is 262 compared to 326 with manual solving. (That's 64 apart ... as a person that counts in the binary system: coincidence ?) For now, I've reached out to 2Captcha support to see if they can resolve the 202 response syntax (jschallenge only) and also to Capsolver to ask whether they have an explanation for what could go wrong. Will update here. |
|
I've got some good news over here. I switched to a different captcha solving service and it ... just works. Attached is an example code that works for me. Please let me know if it works for others here too. import re
import requests
from selenium import webdriver
from seleniumbase import SB
URL = "https://www.immobilienscout24.de/Suche/de/berlin/berlin/wohnung-mieten?numberofrooms=4.0-&price=-3500.0&exclusioncriteria=swapflat&pricetype=rentpermonth&sorting=2&enteredFrom=result_list"
SOLVER_API_ENDPOINT_CREATE = "https://api.capmonster.cloud/createTask"
SOLVER_API_ENDPOINT_GET = "https://api.capmonster.cloud/getTaskResult"
SOLVER_API_KEY = "XYZ"
client = requests.Session()
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
def open_page(sb, url):
sb.driver.uc_open_with_reconnect(url, reconnect_time=2)
def get_aws_cookie(sb):
for cookie in sb.get_cookies():
if (cookie['name'] == 'aws-waf-token'):
return cookie
def resolve_aws(sb, iter=0):
patternJsApi = r'src="([^"]*jsapi\.js)"'
jsapi_matches = re.findall(patternJsApi, sb.driver.page_source)
for match in jsapi_matches:
print(f'SRC Value: {match}')
jsapi = match
patternKey = r'apiKey:\s*"([^"]+)"'
match = re.search(patternKey, sb.driver.page_source)
if match:
api_key = match.group(1)
print(f'apiKey: {api_key}')
else:
print('No apiKey found.')
exit()
data = {
"clientKey": SOLVER_API_KEY,
"task": {
"type": "AmazonTaskProxyless",
"websiteURL": URL,
"userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"captchaScript": jsapi,
"websiteKey": api_key,
"challengeScript": "",
"context": "",
"iv": "",
"cookieSolution": True
}
}
try:
task_id_response = client.post(SOLVER_API_ENDPOINT_CREATE, json=data)
task_id = task_id_response.json()['taskId']
try_cnt=0
while True:
sb.driver.sleep(5)
cookie_response = client.post(SOLVER_API_ENDPOINT_GET, json={"clientKey": SOLVER_API_KEY, "taskId": task_id}).json()
if cookie_response["status"] == "ready":
print(f'ready: {cookie_response}')
cookie = cookie_response["solution"]["cookies"]['aws-waf-token']
old_cookie = get_aws_cookie(sb)
sb.driver.delete_cookie('aws-waf-token')
new_cookie = old_cookie
new_cookie['value'] = cookie
sb.driver.add_cookie(new_cookie)
sb.driver.sleep(3)
sb.driver.refresh()
sb.driver.sleep(5)
return True
elif cookie_response["status"] == "failed":
print(f"solver failed: {cookie_response}")
exit()
else:
print(f"solver not ready yet.... Status: {cookie_response}")
try_cnt+=1
if(try_cnt>5):
print("solver did not process in time for the loop")
exit()
continue
except Exception as e:
print(f"Resolve AWS WAF failed with {e}")
def is_aws_waf(sb):
is_awswaf = re.search("awswaf", sb.driver.page_source)
is_roboter = re.search("Roboter", sb.driver.page_source)
return is_awswaf and is_roboter
with SB(uc=True, headed=True) as sb:
open_page(sb, URL)
if is_aws_waf(sb):
resolve_aws(sb)
if is_aws_waf(sb):
print(".... STILL AWS")
else:
print("Resolved !!!")
else:
exit() |
|
I was eagerly waiting for your post hehe. I can confirm it works for me too🥳 Amazing stuff! (In the name of science I sacrificed 7 dollars since it was the lowest amount I could paypal and didn't find the free trial at first, looks like you can request it from support if you want a free trial without paying - oh well, gonna use it up eventually 😅 ) Looks like scrolling isn't even needed which is great since I only used the SB context for the scroll function. In flathunter the driver gets passed around to different parts of the code and just using that instead of the context might be easier to implement here. |
Since May 2, 2024, there haven't been any ads coming through on my bot that is hosted on Google Cloud. The cloud project is set up exactly as in the tutorial and had been working fine for over a month now. I have logs enabled on the cloud app and every time the application is run, I get the error "IS24 bot detection has identified our script as a bot - we've been blocked", but instead of attempting the captcha with 2captcha as it used to do before, it just closes the application.
Has anyone also encountered this and/or know how to fix it?
The text was updated successfully, but these errors were encountered: