Skip to content

Data extraction tool to get products information from Mary Kay's website. Download images and meta data. Automatic upload your assets to SWS S3 if you want.

License

Notifications You must be signed in to change notification settings

domeniqque-pereira-deel/mk-webscraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MK Web Scraper

A tool to extract products informations of the MK public site.

It will download all the products and meta data like, price, description, colors, and product types in a well structured and versioned .json file (example).

If you want, you'll be able to automatic upload each image and the final .json file to your AWS S3.

I cannot be held responsible for how you will use this tool. Please read the LICENSE file.

This tool was used for study purposes only.

Instalation

First install the dependencies using yarn install. So, in the src/config.ts file, uncomment the product types you want to get the data.

Finally, run yarn start:dev to start the process.

The .json file with all the products will be saved in the output directory.

Automatic Upload

If you want to upload the data to your AWS S3 bucket, you'll need to set your credentials in the .env file and change the IMG_UPLOAD_TO_S3 variable to true.

cp .env.example .env

Before the upload, all images will be optimized using the sharp package.

About

Data extraction tool to get products information from Mary Kay's website. Download images and meta data. Automatic upload your assets to SWS S3 if you want.

Topics

Resources

License

Stars

Watchers

Forks