You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi there,
I'm new to web-crawling and I would like to use puppeteer-cluster and separate tasks into it's own nodejs modules to keep the tasks separate from each other.
/index.js
/tasks/google.js
/tasks/youtube.js
index.js
const{ Cluster }=require('puppeteer-cluster');constgoogleTask=require('./tasks/google.js')constyoutubeTask=require('./tasks/youtube.js')(async()=>{// Create a cluster with 2 workersconstcluster=awaitCluster.launch({concurrency: Cluster.CONCURRENCY_CONTEXT,maxConcurrency: 2,});// Define a task (in this case: screenshot of page)awaitcluster.task(googleTask);awaitcluster.task(youtubeTask);// Shutdown after everything is doneawaitcluster.idle();awaitcluster.close();})();
./tasks/google.js
module.exports=async()=>{awaitpage.goto("https://google.com");// do something}
The text was updated successfully, but these errors were encountered:
4e576rt8uh9ij9okp
changed the title
(Question) Task separation for each website
(Question) Task separated into modules for each website
May 3, 2024
4e576rt8uh9ij9okp
changed the title
(Question) Task separated into modules for each website
(Question) Tasks separated into modules for each website
May 3, 2024
Hi there,
I'm new to web-crawling and I would like to use puppeteer-cluster and separate tasks into it's own nodejs modules to keep the tasks separate from each other.
index.js
./tasks/google.js
The text was updated successfully, but these errors were encountered: