Skip to content

Commit

Permalink
Fix Regencies code in villages districts
Browse files Browse the repository at this point in the history
  • Loading branch information
edwardsamuel committed Nov 13, 2015
1 parent a62ba29 commit 430aba6
Show file tree
Hide file tree
Showing 3 changed files with 15 additions and 20 deletions.
2 changes: 1 addition & 1 deletion README.md
Expand Up @@ -18,7 +18,7 @@ In order to generate new data:
cd scripts
./run.sh

*NOTE*: Please create a MySQL database (named: `indonesia`) and import [`scripts/base_db.sql`](blob/master/scripts/base_db.sql) before run the script. You might need to update MySQL credentials in [`scripts/run.sh`](blob/master/scripts/run.sh).
*NOTE*: Please create a MySQL database (named: `indonesia`) before run the script. You might need to update MySQL credentials in [`scripts/run.sh`](blob/master/scripts/run.sh).

### License

Expand Down
2 changes: 1 addition & 1 deletion scripts/parse.py
Expand Up @@ -50,7 +50,7 @@ def write_data_to_csv(tmp_dir, key):
print 'Writing regencies data...'
write_dict_to_csv(tmp_dir + '/regencies-' + key + '.csv', regencies_dict, 2)
print 'Writing districts data...'
write_dict_to_csv(tmp_dir + '/districts-' + key + '.csv', districts_dict, 5)
write_dict_to_csv(tmp_dir + '/districts-' + key + '.csv', districts_dict, 4)
print 'Writing villages data...'
write_dict_to_csv(tmp_dir + '/villages-' + key + '.csv', villages_dict, 7)
print 'Done.'
Expand Down
31 changes: 13 additions & 18 deletions scripts/run.sh
@@ -1,39 +1,34 @@
#!/bin/sh

DB_NAME=indonesia
DB_USER=root
DB_PASS=root
DB_NAME="indonesia"
MYSQL_CONNECTION_OPTIONS="--user=root --password=root"

TMP_DIR='/tmp/indonesia'
TMP_DIR="/tmp/indonesia"
PWD_DIR=$PWD

if [ ! -d $TMP_DIR ]; then
mkdir -p $TMP_DIR
fi
mkdir -p ${TMP_DIR}

for key in 'a' 'i' 'u' 'e' 'o'; do
echo "Fetching ${key}"
if [ ! -f "$TMP_DIR/$DB_NAME-$key.html" ]; then
curl http://mfdonline.bps.go.id/index.php?link=hasil_pencarian --data "pilihcari=desa&kata_kunci=${key}" > $TMP_DIR/$DB_NAME-$key.html
if [ ! -f "${TMP_DIR}/${DB_NAME}-${key}.html" ]; then
curl http://mfdonline.bps.go.id/index.php?link=hasil_pencarian --data "pilihcari=desa&kata_kunci=${key}" > ${TMP_DIR}/${DB_NAME}-${key}.html
fi

echo "Parsing ${key}"
./parse.py $TMP_DIR $DB_NAME-$key.html $key
./parse.py ${TMP_DIR} ${DB_NAME}-${key}.html ${key}
done
unset key

# mysql -u $DB_USER --password=$DB_PASS $DB_NAME < base_db.sql
mysql $DB_NAME < base_db.sql
mysql ${MYSQL_CONNECTION_OPTIONS} ${DB_NAME} < base_db.sql

for area in 'provinces' 'regencies' 'districts' 'villages'; do
for area in "provinces" "regencies" "districts" "villages"; do
echo "Combining ${area}"
cat $TMP_DIR/$area-*.csv | sort | uniq > ../csv/$area.csv
cat ${TMP_DIR}/${area}-*.csv | sort | uniq > ../csv/${area}.csv

echo "Importing ${area}"
mysqlimport --fields-terminated-by=, --lines-terminated-by="\r\n" -L $DB_NAME ../csv/$area.csv
mysqlimport --fields-terminated-by=, --lines-terminated-by="\r\n" -L ${DB_NAME} ../csv/${area}.csv
done

# mysqldump -u $DB_USER --password=$DB_PASS $DB_NAME > ../mysql/$DB_NAME.sql
mysqldump $DB_NAME > ../mysql/$DB_NAME.sql
mysqldump ${MYSQL_CONNECTION_OPTIONS} ${DB_NAME} > ../mysql/${DB_NAME}.sql

rm -rf $TMP_DIR
# rm -rf $TMP_DIR

0 comments on commit 430aba6

Please sign in to comment.