Skip to content

a collection of web scraping applications: NCEP reanalysis, meteo data from Portugal etc.

Notifications You must be signed in to change notification settings

lhmet/scraping

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

60 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

scraping

A collection of simple scripts that were used in the past for scraping meteorological data from the internet.

Download and process NCEP reanalysis

Something like this will work:

library(scraping)
coor <- data.frame(lon=13.40,lat=52.52)
var <- c('temperature','relative humidity')
years <- c('2000','2001')

request <- def_request(coor,var,years)

get_nc(request)

nc2rds(request)

Variables are available from two datasets. From NCEP such as: temperature,relative humidity,u wind,v wind,soil heat flux,net radiation or precipitation rate. And from GPCC such as gpcc precipitation and number of gauges.

Please have a look at the vignette for more details.

Download ERA-interim (under construction)

In order to download ERA reanalysis you will need to install conda. Please refer to the conda manual to do this in your system.

After installing conda you will need to create an environment containing the python package ecmwf-api-client by typing conda create -n ecmwf ecmwf-api-client. This command will create the environment and install all necessary dependencies.

Then refer to ECMWF to get your own API key and put it into .ecmwfapirc with the form:

{
  "url":"html://api.ecmwf.int/v1",
  "key":"your_key",
  "email":"your_email@xxx"
}

Finally run

def_request()
get_nc()
nc2rds()

as shown previously.

pluviopt

A rvest based package to harvest near real-time pluviometry data from SNIRH/Portugal.

Data is publicly available at SNIRH from the Portuguese Environemntal Agency. Right now it is possible to obtain rainfall data from the telemetry database. Objective is to make it easier to retrieve data from the website directly into an R workflow. Work in progress...

download_hidroweb.py

A selenium-based script for downloading the complete time-series of hundreds of hydro-meteorological stations in the São Francisco river basin.

About

a collection of web scraping applications: NCEP reanalysis, meteo data from Portugal etc.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • R 88.7%
  • Python 11.3%