Skip to content

mottJohn/scrapPath

Repository files navigation

About The Project

Path Grid Data Download

The scripts are used to download path grid data.

There are 5476 grids cover the entire Hong Kong area. Each grid contains meteorology data, concentration data, in 10 levels. Downloading by mouse clicking is definitely not ideal.

Currently, the programs are set to download 2010 meteorology data, 2020 concentration data in Level 1. Users can modify the setting easily by changing dictionary form.

Calculate the Polluatants Concentration Per Grid according to AQOs

heatmap.py

I have reused the code from Aermod to calculate the concentrations of pollutants in each grid according to AQOs requirements. The data will then be plotted as a heatmap in ArcGIS.

Ref:

AQOs

RSP to FSP

Adjustments

Getting Start

  1. Just download the whole thing and unzip
  2. Run each postRequest.py file in anaconda prompt.

To-dos

Scap [Done]

  • Currently, the script use selenium to click the webpage. However, further investigation found that it is actually a form submits to a server. Building a html form as request will be more reliable than clicking webpage, and possibility faster too.

Heatmap

  • The current script runs very slow (An hour) since it loops through all the grips 8 times for all the AQOs requirements. Need to think of some smarter way to do it. (SOLVED: time reduced to 15 mins)

Notes

  • 8 hour average is rolling, but 24 hours averasge is not because we don't know which 8 hours belong to but we know which 24 hours belong to
  • some ambiguity in 0-23 or 1 - 0(24) hour system. The program currently uses 1-0(24) which I am not sure actually. haha.

About

A python script to scrap PATH data and summarise pollutants' concentration according to AQOs

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages