How to run Nodriver on Linux or Lambda?

[raunaqss]

Hi there,

Hope you're having a pleasant day.

Firstly, thanks for making Nodriver and sharing it for all of us to use, it's unbelievably quick and sneaky.

I'm trying to get it working on Linux or AWS Lambda. Using a similar image as this one here for lambda: https://github.com/umihico/docker-selenium-lambda

Here's how my function looks:

async def scrape_with_nodriver(url):
    proxy_server = config('PROXY_SERVER_US')
    chrome_binary_path = config('CHROME_BINARY_PATH', default='/opt/chrome/chrome')
    browser = await nodriver.start(browser_executable_path=chrome_binary_path)
    page = await browser.get(url)
    await page.select('div.element')
    html_source = await page.get_content()
    product_details = parse_product_details(html_source)
    return product_details

However, I'm getting the following error:

{ "errorMessage": "\n ---------------------\n Failed to connect to browser\n ---------------------\n One of the causes could be when you are running as root.\n In that case you need to pass no_sandbox=True \n ", "errorType": "Exception", "requestId": "1d3a2a26-1a2e-4d8b-887d-8375fe504b57", "stackTrace": [ " File \"/var/task/main.py\", line 16, in handler\n return uc.loop().run_until_complete(wayfair_detail_handler.handler(event, context))\n", " File \"/var/lang/lib/python3.12/asyncio/base_events.py\", line 685, in run_until_complete\n return future.result()\n", " File \"/var/task/app/wayfair_detail_handler_uc.py\", line 18, in handler\n product_details = await scrape_with_nodriver(url)\n", " File \"/var/task/app/wayfair_detail_handler_uc.py\", line 27, in scrape_with_nodriver\n browser = await nodriver.start(proxy_server=proxy_server, browser_executable_path=chromedriver_path)\n", " File \"/var/lang/lib/python3.12/site-packages/nodriver/core/util.py\", line 74, in start\n return await Browser.create(config)\n", " File \"/var/lang/lib/python3.12/site-packages/nodriver/core/browser.py\", line 87, in create\n await instance.start()\n", " File \"/var/lang/lib/python3.12/site-packages/nodriver/core/browser.py\", line 343, in start\n raise Exception(\n" ] }

The above message is from AWS Lambda but I'm getting the same error on Linux as well. For linux I've set the browser_executable_path to /opt/google/chrome/chrome (using /opt/google/chrome results in permission errors for some reason).

Any suggestions for how I could get it to work?

Thanks for reading!

Cheers and regards 🫶🏻

[Emoncht]

Try not to run it with root user. I suggest to create a seperate user.

[raunaqss]

I'm going to try this out and get back on whether it worked.

[ganhj99]

@raunaqss Is this working already on your end?