Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timed out waiting for Chrome connection after Restarting Chrome #761

Open
nijinekoyo opened this issue Jul 3, 2023 · 16 comments · May be fixed by #779
Open

Timed out waiting for Chrome connection after Restarting Chrome #761

nijinekoyo opened this issue Jul 3, 2023 · 16 comments · May be fixed by #779

Comments

@nijinekoyo
Copy link

Running prerender in a docker container, after a period of time, all requests Timed out waiting for Chrome connection will appear. After observation, I found that most of them appear after Restarting Chrome. I try to set BROWSER_TRY_RESTART_PERIOD time This problem will appear faster after being shortened. It is speculated that the source of the problem is a problem with the steps to reset Chrome

Version:

$ node server.js
2023-07-03T05:28:50.172Z Starting Prerender
2023-07-03T05:28:50.175Z Starting Chrome
2023-07-03T05:28:50.193Z Prerender server accepting requests on port 3000
2023-07-03T05:28:50.713Z Started Chrome: HeadlessChrome/114.0.5735.198
@nijinekoyo

This comment was marked as outdated.

@AbbasHallal
Copy link

AbbasHallal commented Jul 3, 2023

@nijinekoyo I am having the same issue and the only way to get rid of this is to restart the container as for now

@nijinekoyo
Copy link
Author

@nijinekoyo I am having the same issue and the only way to get rid of this is to restart the container as for now

I'm currently doing the same thing, setting a super long reset chrome time, and then restarting the container regularly, but I always feel that this may cause problems: (

@AbbasHallal
Copy link

@nijinekoyo I followed this solution and it worked for me #755
I reverted back this change. https://github.com/prerender/prerender/pull/748/files
Give it a try!

@nijinekoyo

This comment was marked as outdated.

@jirikrepl
Copy link

jirikrepl commented Jul 11, 2023

Here on GitHub, there are several created issues with 504 errors.

I tried to read the Chrome log file, but there's nothing when a 504 error happens.
I also don't think is somehow related to server resources CPU, and RAM.
I tried to scale the Ubuntu server in the Digital Ocean droplet to the max and got the same 504 error.
I also don't think this is the issue #748
For me, both versions do not work.

I think it might be somehow related either to the OS (e.g. Ubuntu Server on digital ocean),
or Chrome itself, when it crashes with a 504 error.

On the same Ubuntu server droplet, I tried to run this container:
This container works really well, and I am probably going to use it in production.
https://hub.docker.com/r/tvanro/prerender-alpine
https://github.com/tvanro/prerender-alpine

This docker image has older Chrome inside. I checked prerender server diff here on Github,
and it seems there are no major changes (I mean prerender server in the docker image compared with the latest code here).

I suspect the 504 error might be somehow related to newer Chrome, OS, or some environment configuration.
I tried to run prerender server on MacOS, without issue.

views?

@gorkemogut
Copy link

gorkemogut commented Jul 12, 2023

I am trying a workaround by adding the code below:

this.startPrerender();

at the end of server.restartBrowser function in ./lib/server.js:143 to force the restart.

which will be like:

server.restartBrowser = function () {
	this.isBrowserConnected = false;
	util.log(`Restarting ${this.browser.name}`);
	this.browser.kill();
	this.startPrerender(); //<--- This will be added
};

I believe there is an issue with latest chrome versions which causes error on prerender to listen chromeChild.on('close', callback) event and that locks the whole process. That's why triggering the start function without listening close event seems to solve it.

I am not sure if it will cause any memory leak but it is working for now, at least.

Disable browserForceRestart plugin also from server.js on root which relies on close event and probably causing error.

// server.use(prerender.browserForceRestart());

UPDATE:

No chance :( Memory leak as I guessed. Trying to install older Chrome version that I got from the container @jirikrepl shared.

@gorkemogut
Copy link

I am trying a workaround by adding the code below:

this.startPrerender();

at the end of server.restartBrowser function in ./lib/server.js:143 to force the restart.

which will be like:

server.restartBrowser = function () {
	this.isBrowserConnected = false;
	util.log(`Restarting ${this.browser.name}`);
	this.browser.kill();
	this.startPrerender(); //<--- This will be added
};

I believe there is an issue with latest chrome versions which causes error on prerender to listen chromeChild.on('close', callback) event and that locks the whole process. That's why triggering the start function without listening close event seems to solve it.

I am not sure if it will cause any memory leak but it is working for now, at least.

Disable browserForceRestart plugin also from server.js on root which relies on close event and probably causing error.

// server.use(prerender.browserForceRestart());

UPDATE:

No chance :( Memory leak as I guessed. Trying to install older Chrome version that I got from the container @jirikrepl shared.

None of the approaches I tried have worked yet. Even tried chromium versions instead of chrome-stable but no chance. Considering permission issues now.

@breakiboy
Copy link

breakiboy commented Jul 15, 2023

Hi guys,

I faced the same problem. I have for this problem a quick and dirty workaround.👯

create a new file e.q monitor.js and copy paste this:

const { spawn } = require('child_process');

let yourCommand;

function startAndMonitorProcess() {
if (yourCommand) {
yourCommand.stdout.removeAllListeners('data');
yourCommand.stderr.removeAllListeners('data');
yourCommand.kill();
}

yourCommand = spawn('node', ['server.js']);

yourCommand.stdout.on('data', (data) => {
console.log(stdout: ${data});
if (data.includes('Timed out waiting for Chrome connection')) {
console.log("Error detected! Restarting...");
startAndMonitorProcess();
}
});

yourCommand.stderr.on('data', (data) => {
console.error(stderr: ${data});
});

yourCommand.on('close', (code) => {
console.log(child process exited with code ${code});
});
}

startAndMonitorProcess();

this will check the output of server.js if the problem is listed in the console - this script will kill the instance and create a new server.js instance and begins from begin to listen the output.

@gorkemogut
Copy link

gorkemogut commented Jul 16, 2023

Hi guys,

I faced the same problem. I have for this problem a quick and dirty workaround.👯

create a new file e.q monitor.js and copy paste this:

const { spawn } = require('child_process');

let yourCommand;

function startAndMonitorProcess() { if (yourCommand) { yourCommand.stdout.removeAllListeners('data'); yourCommand.stderr.removeAllListeners('data'); yourCommand.kill(); }

yourCommand = spawn('node', ['server.js']);

yourCommand.stdout.on('data', (data) => { console.log(stdout: ${data}); if (data.includes('Timed out waiting for Chrome connection')) { console.log("Error detected! Restarting..."); startAndMonitorProcess(); } });

yourCommand.stderr.on('data', (data) => { console.error(stderr: ${data}); });

yourCommand.on('close', (code) => { console.log(child process exited with code ${code}); }); }

startAndMonitorProcess();

this will check the output of server.js if the problem is listed in the console - this script will kill the instance and create a new server.js instance and begins from begin to listen the output.

Great I've made some additions to put some pm2 magic in it. you can save it as pm2_monitor.js and run it also by using pm2 like pm2 start pm2_monitor.js

const { spawn, exec } = require('child_process');
const pm2 = require('pm2')

let yourCommand;

function pm2Start() {
    if (yourCommand) {
        exec('pkill chrome');
        yourCommand.stdout.removeAllListeners('data');
        yourCommand.kill();
        pm2.disconnect();
    }
    pm2.connect(function (err) {
        if (err) {
            console.error(err)
            process.exit(2)
        }
        pm2.start({
            script: 'server.js',
            name: 'server',
            instances: 7
        }, function (err, apps) {
            if (err) {
                console.error(err)
                return pm2.disconnect()
            }

            yourCommand = spawn('pm2', ['log']);
            yourCommand.stdout.on('data', (data) => {
                if (data.includes('page timed out') || data.includes('parse html timed out') || data.includes('Timed out waiting for')) {
                    console.log("Error detected! Restarting...");
                    pm2.kill(function () {
                        pm2Start();
                    });
                }
            });
        })
    });
}
pm2Start();

@gorkemogut
Copy link

We should find a proper solution for the main issue btw. I am still investigating.

@captainjackrana
Copy link

Same issue.. confirmed on Mac M2 with Chrome 114

@bilipp
Copy link

bilipp commented Aug 23, 2023

Is there any update for this? Facing the same issue

@sovietsolider
Copy link

sovietsolider commented Feb 4, 2024

Facing the same with ubuntu 22.04 and chrome 71

@kruff
Copy link

kruff commented Feb 21, 2024

The same Debian 10 + Chrome 120

@turicas
Copy link

turicas commented Feb 21, 2024

I'm having the same problem on my deployment. The way I partially solved was to run 4 processes (4 containers, each one running node server.js) and reverse proxy a nginx server to them, so if one fails nginx will forward to request to another process.

I'm not sure if it's related to browser restarting (the restart plugin is enabled) but saw this on container's stdout:

 /app/node_modules/ws/lib/websocket.js:988
     const err = new Error(
                 ^

 Error: WebSocket is not open: readyState 2 (CLOSING)
     at sendAfterClose (/app/node_modules/ws/lib/websocket.js:988:17)
     at WebSocket.send (/app/node_modules/ws/lib/websocket.js:405:7)
     at Chrome._enqueueCommand (/app/node_modules/chrome-remote-interface/lib/chrome.js:286:18)
     at /app/node_modules/chrome-remote-interface/lib/chrome.js:88:22
     at new Promise (<anonymous>)
     at Chrome.send (/app/node_modules/chrome-remote-interface/lib/chrome.js:87:20)
     at Object.handler [as continueInterceptedRequest] (/app/node_modules/chrome-remote-interface/lib/api.js:32:23)
     at Chrome.<anonymous> (/app/node_modules/prerender/lib/plugins/blockResources.js:62:30)
     at Chrome.emit (node:events:519:28)
     at Chrome._handleMessage (/app/node_modules/chrome-remote-interface/lib/chrome.js:272:18)

 Node.js v21.6.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants