[OPTIMISATION] fetch platform specific chrome image #518

realityzero · 2024-03-04T18:26:17Z

desc: Environment specific chromium image selection on development environment

vercel · 2024-03-04T18:26:24Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
tiny-helpers	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 14, 2024 8:29am

stefanjudis · 2024-03-05T04:37:22Z

.env.example

@@ -0,0 +1 @@
+NODE_ENV=development


Should we make development the default and change the conditions to detect the production environment instead? Then we wouldn't need the env file locally.

Done. That's a nice catch since there are no other envs we can default to dev.

stefanjudis · 2024-03-05T04:38:37Z

README.md

@@ -252,6 +252,13 @@ npm run dev
                <sub><b>Gonçalo Morais</b></sub>
            </a>
        </td>
+        <td align="center">


FYI — this readme section was broken for a while and when we're at it, i'll probably remove it because it doesn't show all contributoes.

Your avatar will still be shown on tiny-helpers.dev though.

stefanjudis · 2024-03-05T04:39:12Z

api/screenshot.js

+    console.log('exePath:', exePath);
+    if (isDev) {
+        options = {
+        args: [],


~~Is there a reason why args and defaultViewport are locally different than in prod?~~

Ahh. I see — because chromium isn't available. Could we still add --hide-scrollbars and --disable-web-security? 🤔

stefanjudis · 2024-03-05T04:42:00Z

@realityzero Welcome to TH and thank you so much for contributing. 💯

This topic has been on the list for ages so I'm exited that someone is tackling it. I've left some comments and on the way and will test is locally once we discussed these.

Thanks again and have a great day!

realityzero · 2024-03-13T13:38:32Z

@realityzero Welcome to TH and thank you so much for contributing. 💯

This topic has been on the list for ages so I'm exited that someone is tackling it. I've left some comments and on the way and will test is locally once we discussed these.

Thanks again and have a great day!

woahh @stefanjudis this made my day

stefanjudis · 2024-03-15T07:43:01Z

@realityzero I just checked out the branch and have run vercel dev (I should document this 🤦‍♂️ ).

Unfortunately, it's not working locally and I'm running in target closed errors.

Error log


> Ready! Available at http://localhost:3000
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
TargetCloseError: Protocol error (Page.captureScreenshot): Target closed
    at CallbackRegistry.clear (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Connection.ts:186:30)
    at CDPSessionImpl._onClosed (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Connection.ts:611:21)
    at Connection.#onClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Connection.ts:363:15)
    at WebSocket. (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/NodeWebSocketTransport.ts:60:22)
    at callListener (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:220:9)
    at WebSocket.emit (node:events:517:28)
    at WebSocket.emit (node:domain:489:12)
    at WebSocket.emitClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:258:10)
    at Socket.socketOnClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:1264:15)
Error: Navigation failed because browser has disconnected!
    at new LifecycleWatcher (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/LifecycleWatcher.ts:109:11)
    at Frame.goto (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Frame.ts:108:21)
    at CDPPage.goto (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Page.ts:923:35)
    at getScreenshot (/Users/stefanjudis/Projects/github.com/tiny-helpers/api/screenshot.js:62:14)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async module.exports (/Users/stefanjudis/Project
    ...
    ...

What's your local setup to test this?

realityzero · 2024-03-21T08:02:28Z

@stefanjudis Apologies for getting onto this quite late. I couldn't test the solution end to end locally from tiny-helpers project. However, I followed the same abstracted approach of picking platform specific chromium in one of my recent projects: ScrapifyX.

That's my bad.

Could you please try this out: Stackoverflow

Or just tell me how to invoke screenshot.js after running this project. I was unable to find any references from UI.

stefanjudis · 2024-03-25T07:24:10Z

@realityzero

Apologies for getting onto this quite late.

No worries. I'm still very grateful that you're taking this on! 🎉

Or just tell me how to invoke screenshot.js after running this project.

That's my bad. As this project is deployed to vercel I run it via the CLI which I have installed globally.

$ vercel dev
Vercel CLI 33.6.0
> Running Dev Command “npx @11ty/eleventy --serve --watch --port $PORT”
Writing _site/index.html from ./site/index.html.
Writing _site/home/latest/index.html from ./site/tag-sorted-by-date.njk.
Writing _site/home/index.html from ./site/tag.njk.
...
...
[Browsersync] Access URLs:
 ----------------------------------------
       Local: http://localhost:3000
    External: http://192.168.178.88:3000
 ----------------------------------------
          UI: http://localhost:3001
 UI External: http://localhost:3001
 ----------------------------------------
[Browsersync] Serving files from: _site
> Ready! Available at http://localhost:3000
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
TargetCloseError: Protocol error (Page.captureScreenshot): Target closed
    at CallbackRegistry.clear (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/CallbackRegistry.ts:93:30)
    at CdpCDPSession._onClosed (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/cdp/CDPSession.ts:149:21)
    at Connection.#onClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/cdp/Connection.ts:205:15)
...

I also looked briefly into this and no SO magic helped out for the local screenshoting. :/

realityzero · 2024-03-29T12:54:45Z

@stefanjudis
Whenever I've read TargetCloseError or Target closed in Puppeteer, it is almost always a memory issue. Can you try adding adding following args:

Let's just add this first --disable-dev-shm-usage
If it still breaks, add one more argument --shm-size=3gb

Ref:

puppeteer/puppeteer#1790
puppeteer/puppeteer#1175

stefanjudis · 2024-03-29T15:46:11Z

@realityzero Unfortunately, neither of the options work.

I'm still running into the same exceptions. Did you try the Vercel CLI to see how to tackle it?

realityzero · 2024-04-07T09:55:02Z

Hey @stefanjudis ! Apologies man for such a late reply. Certain life challenges.

I did some debugging locally in two different stages.

Look at this diff (github wasn't allowing me to upload a .patch file)

diff --git a/api/screenshot.js b/api/screenshot.js
index c285f0c..8be5f87 100644
--- a/api/screenshot.js
+++ b/api/screenshot.js
@@ -27,7 +27,7 @@ async function getOptions(isProd) {
         options = {
           args,
           executablePath: exePath,
-          headless: 'new',
+          headless: false,
         };
     }
     return options;
@@ -57,7 +57,7 @@ function checkUrl(string) {
   return true;
 }
 
-export async function getScreenshot(url, ratio = 1) {
+async function getScreenshot(url, ratio = 1) {
   const page = await getPage();
   await page.goto(url, {
     waitUntil: 'domcontentloaded',
@@ -68,6 +68,8 @@ export async function getScreenshot(url, ratio = 1) {
     devicePixelRatio: ratio,
   });
   const file = await page.screenshot();
+  const base64Image = new Buffer(file).toString('base64');
+  console.log('done w/ SS:', base64Image);
   return file;
 }
 
@@ -92,3 +94,6 @@ module.exports = async (req, res) => {
       );
   }
 };
+
+
+getScreenshot('https://news.ycombinator.com/item?id=39894820#39916444', 1);
\ No newline at end of file

I simply executed node screenshot.js. Here I was able to get a base64 and used some base64 to image converted online and it went fine.
2. I tried vercel dev command to replicate the same connection level thing of deployment. However I couldn't navigate to a page that takes screenshots. I might need help in that. But it triggered screenshot.js behind the scenes in terminal and executed same thing as point 1.

How about we get over a call for this in the coming week?

stefanjudis · 2024-05-13T18:12:26Z

@realityzero Hey, so sorry for letting this slip through. :/ Life's pretty wild on my end.

But in the meantime, something changed on vercel's side and things broke entirely. I think I'll move the screenshots off serverless fns. It's just to briddle. ;/

Sorry for wasting your time.

realityzero added 2 commits March 4, 2024 23:37

fetch platform specific chrome image

bf8d046

contributors update

bbc7893

vercel bot deployed to Preview March 4, 2024 18:26 View deployment

stefanjudis reviewed Mar 5, 2024

View reviewed changes

realityzero added 2 commits March 14, 2024 13:51

chrome same args for dev

322f1c6

env check defaults to dev

ee693dd

vercel bot deployed to Preview March 14, 2024 08:29 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OPTIMISATION] fetch platform specific chrome image #518

[OPTIMISATION] fetch platform specific chrome image #518

realityzero commented Mar 4, 2024

vercel bot commented Mar 4, 2024 •

edited

stefanjudis Mar 5, 2024

realityzero Mar 14, 2024

stefanjudis Mar 5, 2024

stefanjudis Mar 5, 2024 •

edited

realityzero Mar 14, 2024

stefanjudis commented Mar 5, 2024

realityzero commented Mar 13, 2024

stefanjudis commented Mar 15, 2024

realityzero commented Mar 21, 2024

stefanjudis commented Mar 25, 2024

realityzero commented Mar 29, 2024

stefanjudis commented Mar 29, 2024

realityzero commented Apr 7, 2024

stefanjudis commented May 13, 2024 •

edited

		@@ -0,0 +1 @@
		NODE_ENV=development

[OPTIMISATION] fetch platform specific chrome image #518

Are you sure you want to change the base?

[OPTIMISATION] fetch platform specific chrome image #518

Conversation

realityzero commented Mar 4, 2024

vercel bot commented Mar 4, 2024 • edited

stefanjudis Mar 5, 2024

Choose a reason for hiding this comment

realityzero Mar 14, 2024

Choose a reason for hiding this comment

stefanjudis Mar 5, 2024

Choose a reason for hiding this comment

stefanjudis Mar 5, 2024 • edited

Choose a reason for hiding this comment

realityzero Mar 14, 2024

Choose a reason for hiding this comment

stefanjudis commented Mar 5, 2024

realityzero commented Mar 13, 2024

stefanjudis commented Mar 15, 2024

realityzero commented Mar 21, 2024

stefanjudis commented Mar 25, 2024

realityzero commented Mar 29, 2024

stefanjudis commented Mar 29, 2024

realityzero commented Apr 7, 2024

stefanjudis commented May 13, 2024 • edited

vercel bot commented Mar 4, 2024 •

edited

stefanjudis Mar 5, 2024 •

edited

stefanjudis commented May 13, 2024 •

edited