Uses Selenium and Chrome driver to open webpages in a headless (invisible) browser and access their contents
Currently scrapable: Amazon India, Flipkart, BigBasket
Required on your system:
- Python (added to PATH): Install from https://www.python.org/downloads/ and add to PATH variable
- Chrome: Install from https://www.google.com/intl/en_us/chrome/
- Chrome Driver: Download (same version as Chrome!) from https://chromedriver.chromium.org/downloads (versions <= 114) or https://googlechromelabs.github.io/chrome-for-testing (versions >= 115)
- Selenium module for Python: Run command
pip install selenium
Windows users can quickly run by clicking on RUN.bat
Otherwise, run commands
cd path/to/WebScraper
and python main.py
If cloned with Git, Windows users can use UPDATE.bat to pull the latest version while preserving consts.txt
Selenium handshake failure errors can mostly be ignored