This project is a web crawler that retrieves items from CSGO Steam Market and stores them in a Mongo Database.
Items available: Knives • Pistols • Rifles • Shotguns • Snipers • Machineguns • Smgs • Gloves
-
Set up a MongoDB database, I used a free tier Atlas MongoDB.
-
Get a Steam API Key, following these instructions.
-
Use package manager pip to install all needed libs.
bash pip install -r requirements.txt -
Create the file .env in the project root folder with the following content:
STEAM_API_KEY='PLACE YOU STEAM API KEY HERE'
LAST_PROCESSED_PAGE_KNIFE='-1'
LAST_PROCESSED_PAGE_PISTOL='-1'
LAST_PROCESSED_PAGE_RIFLE='-1'
LAST_PROCESSED_PAGE_SHOTGUN='-1'
LAST_PROCESSED_PAGE_SNIPER='-1'
LAST_PROCESSED_PAGE_MACHINEGUN='-1'
LAST_PROCESSED_PAGE_SMG='-1'
LAST_PROCESSED_PAGE_GLOVE='-1'
BATCH_SIZE=50
LAST_PROCESSED_PAGE='1'
DATABASE_PASSWORD='PLACE YOUR DATABASE PASSWORD HERE'
DATABASE_USER='PLACE YOUR DATABASE PASSWORD HERE'
DATABASE_URL='mongodb+srv://USER:PASSWORD@HOST/DATABASE?retryWrites=true&w=majority'
LOG_LEVEL='INFO'
To run this project is very simple, you need to run the crawler script.
python crawler.py
It will crawl through Steam Market and store the items in the collection named item with the following structure:
{
"name":"★ Bowie Knife | Case Hardened (Well-Worn)",
"type":"weapon",
"subtype":"knife",
"game_type":"★ Covert Knife"
}
There are two possible types of the items:
- Weapon
- Hands
And the following subtypes:
- Weapon Type
- Knife
- Pistol
- Rifle
- Shotgun
- Sniper
- Machinegun
- Smg
- Hands Type
- Glove
The crawler script creates the environment variables from the .env file and uses them to control the pages of the market for each item subtype.
It's important to notice that the Steam Web API blocks the access for some time if many requests are made, so the script will wait a random time between 10 and 30 seconds.
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
CSGO png image for the project icon csgo-icon. Credits to Ahkâm.
Crawler png image for the project icon crawler-icon. Credits to Pixelmeetup.