Skip to content

lucasxlu/LagouJob

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

71 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Data Analysis of Lagou Job

Lagou

Introduction

This repository holds the code for job data analysis of Lagou. The main functions included are listed as follows:

  1. Crawling job data from Lagou, and get the latest information of jobs about Internet.
  2. Proxies are collected from XiCiDaiLi.
  3. Data analysis and visualization.
  4. Crawling job details info and generate word cloud as Job Impression.
  5. In order to train a NLP task with machine learning, the data of interviewee's comments will be stored in mongodb

Prerequisites

  1. Install 3rd party libraries

    sudo pip3 install -r requirements.txt

  2. Install mongodb and start mongodb service [optional]

    sudo service mongod start

How to Use

  1. clone this project from github.
  2. Lagou's anti-spider strategy has been upgrade frequently recently. I suggest you run proxy_crawler.py to get IP proxies and execute the code with PhantomJS.
  3. run m_lagou_spider.py to crawl job data, it will generate a collection of Excel files in ./data directory.
  4. run hot_words_generator.py to cut sentences, it will return TOP-30 hot words and wordcloud figure.

Analysis Results

Image1
Image2
Image3
Image4
Image5
Image6
Image7

Report

  • For technical details, please refer to my answer at Zhihu.
  • The PDF report can be downloaded from here.

Change Log

  • [V2.0] - 2019.04. Upgraded to PhantomJS and IP proxies.
  • [V1.2] - 2017.05. Rewrite WordCloud visualization module.
  • [V1.0] - 2017.04. Upgraded to mobile Lagou.
  • [V0.8] - 2016.05. Finish Lagou PC web spider.

LICENSE

Apache-2.0