Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DynamicScraper closeFactory() seems to have memory leak #25

Open
ahashp opened this issue Jan 14, 2015 · 1 comment
Open

DynamicScraper closeFactory() seems to have memory leak #25

ahashp opened this issue Jan 14, 2015 · 1 comment

Comments

@ahashp
Copy link

ahashp commented Jan 14, 2015

I'm running nodeJS v0.10.35 on Ubuntu 14.04.1 LTS.
I read your documentation and calls

  • scraperjs.DynamicScraper.startFactory()

before I request multiple pages in parallel to scrape.
On completion of all the scraping, (using Promise). I would call

  • scraperjs.DynamicScraper.closeFactory()

However, looking at the memory usage, after a few hundred rounds of function calls. I see there are a bunch of processes belonging to phantomjs sitting there using up memory, eventually the system runs out of memory (4G) and nodeJS crashes.

I looked through the source code for this PhantomPoll class, I don't see any where it close the "Page", is this close() for each page needed to release the memory? could this be the reason for the memory "leak" that I see? Could you please spend a little bit of your time to help check? much appreciated.

@ruipgil
Copy link
Owner

ruipgil commented Jan 14, 2015

I think the leak is not directly from the scraper/node process, but rather a mismanagement of the phantom pages. I had some issues with it in the past.
I'll take a look into it in the next couple of days. Thanks for your input.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants