Take full page screenshots #143

[kamtschatka]

Added the fullPage flag to take full screen screenshots updated the UI accordingly to properly show the screenshots instead of scaling it down

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
hoarder-app-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 10, 2024 9:21pm
hoarder-app-landing	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 10, 2024 9:21pm

[MohamedBassem]

First of all, again thanks a lot for taking the time and implementing that feature you requested! :) I tried it locally, but it seems that for some reason, sometimes, it results into the crawler getting stuck fetching that page.

for example when fetching that link, it always times out which doesn't happen when full page screenshot is enabled:

2024-05-11T17:33:20.086Z info: [Crawler][279] Will crawl "https://www.amazon.co.uk/ASUS-GeForce-Graphics-DisplayPort-TUF-RTX4090-O24G-GAMING/dp/B0BGV6LQYR/ref=pd_ci_mcx_mh_mcx_views_0?pd_rd_w=HX8NY&content-id=amzn1.sym.c428a25c-eebb-4a80-982c-a8845b70a765:amzn1.symc.ca948091-a64d-450e-86d7-c161ca33337b&pf_rd_p=c428a25c-eebb-4a80-982c-a8845b70a765&pf_rd_r=KEX284MKDMNSA32NBMFC&pd_rd_wg=em7oq&pd_rd_r=da88c671-7bc3-488b-999d-e650f5af151e&pd_rd_i=B0BGV6LQYR" for link with id "hs653stzi2oqtleqsvjla4wb"
2024-05-11T17:33:25.181Z info: [Crawler][279] Successfully navigated to "https://www.amazon.co.uk/ASUS-GeForce-Graphics-DisplayPort-TUF-RTX4090-O24G-GAMING/dp/B0BGV6LQYR/ref=pd_ci_mcx_mh_mcx_views_0?pd_rd_w=HX8NY&content-id=amzn1.sym.c428a25c-eebb-4a80-982c-a8845b70a765:amzn1.symc.ca948091-a64d-450e-86d7-c161ca33337b&pf_rd_p=c428a25c-eebb-4a80-982c-a8845b70a765&pf_rd_r=KEX284MKDMNSA32NBMFC&pd_rd_wg=em7oq&pd_rd_r=da88c671-7bc3-488b-999d-e650f5af151e&pd_rd_i=B0BGV6LQYR". Waiting for the page to load ...
2024-05-11T17:33:29.020Z info: [Crawler][279] Finished waiting for the page to load.
2024-05-11T17:34:20.102Z error: [Crawler][279] Crawling job failed: Error: Timed-out after 60 secs
2024-05-11T17:34:22.139Z info: [Crawler][279] Will crawl "https://www.amazon.co.uk/ASUS-GeForce-Graphics-DisplayPort-TUF-RTX4090-O24G-GAMING/dp/B0BGV6LQYR/ref=pd_ci_mcx_mh_mcx_views_0?pd_rd_w=HX8NY&content-id=amzn1.sym.c428a25c-eebb-4a80-982c-a8845b70a765:amzn1.symc.ca948091-a64d-450e-86d7-c161ca33337b&pf_rd_p=c428a25c-eebb-4a80-982c-a8845b70a765&pf_rd_r=KEX284MKDMNSA32NBMFC&pd_rd_wg=em7oq&pd_rd_r=da88c671-7bc3-488b-999d-e650f5af151e&pd_rd_i=B0BGV6LQYR" for link with id "hs653stzi2oqtleqsvjla4wb"
2024-05-11T17:34:24.868Z info: [Crawler][279] Successfully navigated to "https://www.amazon.co.uk/ASUS-GeForce-Graphics-DisplayPort-TUF-RTX4090-O24G-GAMING/dp/B0BGV6LQYR/ref=pd_ci_mcx_mh_mcx_views_0?pd_rd_w=HX8NY&content-id=amzn1.sym.c428a25c-eebb-4a80-982c-a8845b70a765:amzn1.symc.ca948091-a64d-450e-86d7-c161ca33337b&pf_rd_p=c428a25c-eebb-4a80-982c-a8845b70a765&pf_rd_r=KEX284MKDMNSA32NBMFC&pd_rd_wg=em7oq&pd_rd_r=da88c671-7bc3-488b-999d-e650f5af151e&pd_rd_i=B0BGV6LQYR". Waiting for the page to load ...
2024-05-11T17:34:25.871Z info: [Crawler][279] Finished waiting for the page to load.
2024-05-11T17:35:22.152Z error: [Crawler][279] Crawling job failed: Error: Timed-out after 60 secs

I don't know how puppeteer implements this, but I wonder if it misbehaves if the page expands on scrolling for example or something.

[kamtschatka]

Thanks, I'll check it out.

[kamtschatka]

Are you running that with the docker container for chrome or just with the worker? I tried that out many times now and never had any issues.

[MohamedBassem]

Ok, no worries. Let's merge it and I'll add some protections so that taking screenshots timing out doesn't fail the entire crawling :)