This is a "meta-repository" for Open Climate Fix's solar electricity nowcasting project. See this great Wired article about OCF's solar electricity forecasting work for a good intro to solar electricity nowcasting.
The plan is to enable the community to build the world's best near-term forecasting system for solar electricity generation, and then let anyone use it! :) We'll do this by using state-of-the-art machine learning and 5-minutely satellite imagery to predict the movement of clouds over the next few hours, and then use this to predict solar electricity generation.
The term "nowcasting" just means "forecasting for the next few hours using statistical techniques".
In OCF, we're curious to see if it's possible to rapidly mitigate climate change by:
- Enabling thousands of people to help solve ML problems which, if solved, might help reduce CO2 emissions
- Running small(ish) pilot projects to implement the best solution in industry
- Enabling thousands of practitioners to use the code in their products.
It's really, really, really hard to estimate climate impact of forecasting! But, as a super-rough back-of-the-envelope calculation, we estimate that better solar forecasts, if rolled out globally, could reduce CO2 emissions by about a billion tonnes between now and 2035.
- List of "good first issues": GitHub "issues" which describe changes we'd like to make to the code.
- OCF's coding style
- The main tools we use include: PyTorch, PyTorch Lighting, xarray, pandas, pvlib
- nowcasting_dataset: Pre-prepares ML training batches. Loads satellite data, numerical weather predictions, solar PV power generation timeseries, and other datasets. Outputs pre-prepared ML training batches as NetCDF files (one batch per NetCDF file).
- Satip: Retrieve, transform and store EUMETSAT data.
- pvoutput: Python code for downloading PV data from PVOutput.org.
- satellite_image_processing
- eumetsat: Tools for downloading and processing satellite images from EUMETSAT
| Repo | Description | Main Developer |
|---|---|---|
| UK PV National XG | National GSP PV forecasting using Gradient Boosted Methods. | Peter Dudfield |
| Pseudo Labeller | Pseudo Labeller for generating training labels for other PV generation forecasting | Jacob Bieker |
| PVnet | PV net main repo | James Fulton |
| PV Site Prediction | ML experiments and models for SBRI project | Zak Watts |
| Satflow | Satellite Optical Flow with machine learning models. Predicting the next few hours of satellite imagery from the recent history of satellite imagery (and other data sources). | Jacob Bieker |
| Predict PV Yield | Using optical flow (and the output of satflow) & machine learning to predict solar PV yield (i.e. to predict the power generated by solar electricity systems over the next few hours). An older set of experiments is in predict_pv_yield_OLD, which is no longer maintained. | Jack Kelly |
| Nowcasting Utils | Forecasting performance metrics, plotting functions, loss functions, etc. | Peter Dudfield |
| Nowcasting Dataloader | PyTorch dataloader for taking pre-prepared batches from nowcasting-dataset and getting them into our models. |
Jacob Bieker |
| Nowcasting Dataset | Prepare batches of data for training machine learning solar electricity nowcasting data | Jack Kelly |
| Repo | Description | Main Developer |
|---|---|---|
| MetNet | PyTorch Implementation of Google Research's MetNet (Sønderby et al. 2020), inspired from Thomas Capelle's metnet_pytorch. | Jacob Bieker |
| Skillful Nowcasting | Implementation of DeepMind's Skillful Nowcasting GAN (Ravuri et al. 2021) in PyTorch Lightning. | Jacob Bieker |
| Perceiver Pytorch | Implementation of DeepMind's Perceiver (Jaegle et al. 2021) and Perceiver IO (Jaegle et al. 2021) in Pytorch. Forked from lucidrains/perceiver-pytorch | Jack Kelly |
| Graph Weather | PyTorch implementation of Ryan Keisler's 2022 "Forecasting Global Weather with Graph Neural Networks" paper (https://arxiv.org/abs/2202.07575) | Jacob Bieker |
| Repo | Description | Main Developer |
|---|---|---|
| PVoutput | Python code for downloading PV data from PVOutput.org | Jack Kelly |
| GSP Consumer | Collect GSP solar generation data from PVLive | Peter Dudfield |
| PV Consumer | Consumer PV data from various sources | Peter Dudfield |
| MetOfficeDataHub | Python wrapper around MetOffice Atmospheric Model Data REST API | Peter Dudfield |
| NWP consumer | Microservice for consuming NWP data. | Sol Cotton |
| UK PV National GSP API | API for hosting nowcasting solar predictions. Will just return 'dummy numbers' until about mid-2022! | Peter Dudfield |
| Satip | Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data | Jacob Bieker |
| Nowcasting Forecast | Making live forecasts for the nowcasting project | Peter Dudfield |
| Nowcasting Datamodel | Datamodel for the nowcasting project | Peter Dudfield |
| Nowcasting Metrics | Repo to automatically run metrics on the nowcasting forecast | Peter Dudfield |
| Nowcasting Alerts Cron | Nowcasting Alerts cron Worker | Brad Fulford |
| Nowcasting | Front End repo for the Nowcasting project. | Brad Fulford |
| UK Analysis Dashboard | This is a Streamlit app for the OCF team that reports database statistics | Rachel Tipton |
| Nowcasting | Front End repo for the Nowcasting project. | Brad Fulford |
| OCF Infrastructure | Infrastructure code for OCF's cloud environments | Sol Cotton |
| PV Site Production | Production service for PV site level forecasts | Peter Dudfield |
| PV Site API | Site specific API for SBRI project | Peter Dudfield |
| PV Site Mobile | Front End web application for site-level forecast UI | Brad Fulford |
| PV Site Datamodel | Datamodel for PV sites | Peter Dudfield |
| OCF Datapipes | OCF's DataPipe based dataloader for training and inference | Jacob Bieker |
| Repo | Description | Main Developer |
|---|---|---|
| NWP | Tools for downloading and processing numerical weather predictions | Jacob Bieker |
| GFS Downloader | NCEP GFS 0.25 Degree Global Forecast Grids Historical Archive: https://rda.ucar.edu/datasets/ds084.1/ | Zak Watts |
| OCF ML Metrics | Collection of simple baseline models and metrics for standardized evaluation of OCF forecasting models | Jacob Bieker |
| OCF Blocs2 | Blosc2 codec used for OCF's Zarr compression | Jacob Bieker |
| .github | Various Community Health Files | Jacob Bieker |
| Solar and Storage | Solar and Storage optimization code | Peter Dudfield |
For a complete list of all of OCF's repositories tagged with "nowcasting", see this link
Thanks goes to these wonderful people (emoji key):
 Damien Tanner 📆 |
 lina 💻 |
 AlaaTohamy 💻 |
 Flo 💻 |
 dantravers 🤔 |
 Peter Dudfield 💻 |
 braddf 💻 |
 rachel tipton 👀 💻 |
This project follows the all-contributors specification. Contributions of any kind welcome!
