News-Scraper

A Python script to download news from a collection of news sources. Written with Scrapy.

Also included is a script to perform LDA on the downloaded articles.

Spiders Provided:

To Use:

scrapy crawl [ spider-name ] [ -o output-file ] [ -t output-file-type ]

Using the csv output from previous step, use:

simple_lda.py -i FILENAME [-s {none,porter,porter2}] [-ni NUM_ITER] [ -twc TOPWORDS_COUNT ]

Name		Name	Last commit message	Last commit date
Latest commit History 78 Commits
.ipynb_checkpoints		.ipynb_checkpoints
allwords		allwords
build/lib/news_scraper		build/lib/news_scraper
datasets		datasets
dictionaries		dictionaries
executables		executables
images		images
lib		lib
models		models
news_scraper		news_scraper
project.egg-info		project.egg-info
tagged_datasets		tagged_datasets
.DS_Store		.DS_Store
.gitignore		.gitignore
Document Clustering.ipynb		Document Clustering.ipynb
LDaVis.ipynb		LDaVis.ipynb
Procfile		Procfile
README.md		README.md
Untitled.ipynb		Untitled.ipynb
doc_cluster.pkl		doc_cluster.pkl
doc_cluster.pkl_01.npy		doc_cluster.pkl_01.npy
doc_cluster.pkl_02.npy		doc_cluster.pkl_02.npy
l.php?u=https%3A%2F%2Fcdn.fbsbx.com%2Fv%2Ft59.2708-21%2F13104659_10153989009453673_1892764892_n.pptx%2FIsu-isu-Cabaran-semasa-tamadun-islam_Mei16.pptx%3Foh%3D8f6a354778718029a788df35123d7dc3%26oe%3D574AC068%26dl%3D1		l.php?u=https%3A%2F%2Fcdn.fbsbx.com%2Fv%2Ft59.2708-21%2F13104659_10153989009453673_1892764892_n.pptx%2FIsu-isu-Cabaran-semasa-tamadun-islam_Mei16.pptx%3Foh%3D8f6a354778718029a788df35123d7dc3%26oe%3D574AC068%26dl%3D1
log.txt		log.txt
log_soya_6months.txt		log_soya_6months.txt
log_soya_6months_contents.txt		log_soya_6months_contents.txt
powtoon.csv		powtoon.csv
requirements.txt		requirements.txt
scrapinghub.yml		scrapinghub.yml
scrapy.cfg		scrapy.cfg
setup.py		setup.py
simple_lda.py		simple_lda.py
soya_6_month_clusters_contents.txt		soya_6_month_clusters_contents.txt
soya_6_month_clusters_titles.txt		soya_6_month_clusters_titles.txt

HiiYL/News-Scraper