This is a amazon scraper build using selenium it can scrape product data from the website and write it to a csv file i have used selenium because it resembles a bit like human and it is a advanced testing frameworks which offers many advantages for scraping and i also used beautiful soup to extract data from the scraped data
1.The code is same for both windows and linux you need to install seperate chromedriver for seperate environments you can install chromedriver at https://chromedriver.chromium.org/downloads The chromedriver file should be placed at root folder
2.setup a python environment using the command python -m venv myenv
3.activate the python virtual environment for windows myenv/Scripts/activate.ps1 and for linux source myenv/bin/activate
4.install the python modules in the python virtual environment using the command pip install -r requirements.txt
- simply run the project in windows
python amazon_scraper.pyand in linuxpython3 amazon_scraper.py1.To run project in windows run the commandpython amazon_scraper.py2.To run the project in linux run the commandpython3 amazon_scraper.py
1.When you first run the project the selenium will scrape the product and write each product source code into html fiels in the products directory
2.In the next step the data is extracted from the html files and written to output.csv file
1.amazon-scraper.py is the main python file which scrapes and writes data to the csv files
2.requirements.txt is the file which contains all the modules required by the project to function without errors
3.automate.bat is used to push code to github in windows
4.automate.sh is used to push code to github in linux
https://drive.google.com/file/d/1xS28dAszifytomf69G2MfKjBfMia_it8/view?usp=sharing
This project loves to accept contributions from everyone
- HTML
- CSS
- JavaScript
For support, you can buy me a coffee
This project license is MIT LICENSE