A curated list of datasets, tools and papers for machine learning in heliophysics.
The primary goal of awesome-helio is to provide a compiled list for all the great resources contributing to the state of the art in heliophysics that are out there but are often hard to find. awesome-helio is mainly focussed on SDO and IRIS data and targets machine learning groups but can also provide important insights for other users.
Please take a quick look at the contribution guidelines first.
If you find any datasets, tools and papers that is missing or is not a good fit, please submit a pull request to improve this file. Thank you!
SDO and IRIS are part of a larger fleet of spacecraft strategically placed throughout our heliosphere:
Heliophysics Mission Fleet, Source NASA
A comprehensive list of different missions can be found here.
- Machine Learning, Statistics, and Data Mining for Heliophysics - Bobra, Monica, and James Paul Mason. "Machine learning, statistics, and data mining for heliophysics."
- Machine learning techniques for space weather - Camporeale, Enrico, Simon Wing, and Jay Johnson, eds. "Machine learning techniques for space weather." Elsevier, 2018.
- Deep Learning in Solar Astronomy - Xu, Long, Yihua Yan, and Xin Huang. "Deep Learning in Solar Astronomy."
For datasets at FHNW refer to this link.
JPEG vs. Fits (Flexible Image Transport System - commonly used digital file format in astronomy)
- e-Callisto Data access - The CALLISTO spectrometer is a programmable heterodyne receiver used for observation of solar radio bursts for astronomical science, education, outreach and citizen science as well as rfi-monitoring.
- LMSAL data search - Useful search interface by Lockheed Martin with links to various resources for each observation. Use our internal mirror for data download or work directly on one of the dedicated machines.
- IRISdataset (Currently only for internal access) - Dataset with selected representative observations for different observation classes.
TODO
Before working with SDO data make sure to checkout the Guide to SDO Data Analysis, which contains a set of instructions on how to browse, find, download, and analyze SDO data.
- Curated Image Parameter Dataset - Massive image parameter dataset extracted from the Solar Dynamics Observatory (SDO) mission's AIA instrument, for the period of January 2011 through the current date, with the cadence of six minutes, for nine wavelength channels.
- JSOC - Data products from the Solar Dynamics Observatory, as well as certain other missions and instruments, are available from the JSOC database.
- DeepSDO Event dataset - Dataset curated by experts containing three solar event categories (coronal holes, sunspots, and prominences). Suitable for object detection using deep learning-based models.
- LSDO - A Large-scale Solar Dynamics Observatory image dataset for computer vision applications Dataverse.
- Machine Learning Data Set for NASA's Solar Dynamics Observatory - A curated dataset from the NASA Solar Dynamics Observatory (SDO) mission in a format suitable for machine learning research.
- Machine Learning Data Set for NASA's Solar Dynamics Observatory - not corrected for degradation - Version of the SDO ML v1 dataset not corrected for degradation over time.
- Machine Learning Data Set for NASA's Solar Dynamics Observatory v2 - The second version of the curated dataset from the NASA Solar Dynamics Observatory (SDO) mission in a format suitable for machine learning research. colab intro code
- SDOBenchmark - Machine learning image dataset for the prediction of solar flares.
- SOHO/MDI & SDO/HMI line-of-sight Magnetogram Dataset - Curated dataset consisting of co-aligned, co-temporal observations of the same physical structures as observed by HMI and MDI (rotated to the corresponding the HMI frame), ideal for learning-based super-resolution techniques. colab code intro
- SOHO/SDO ML Ready Dataset - Code to generate and temporally sync SoHO and/or SDO Mission image products to make a standardized machine-learning-ready dataset.
- Solar Flare Prediction from Time Series of Solar Magnetic Field Parameters - Processed dataset provided for the IEEE Big Data 2019 Big Data Cup consisting of a set of magnetic field parameters calculated from individual SHARPs.
- SWAN-SF Multivariate time series (MVTS) data extracted from HMI Active Region Patch (SHARP) series.
When working with AIA data and analyzing short-lived phenomena, make sure to consider respiking the data as the AIA despiking algorithm might remove some of the features (Young, Peter R., et al. "An Analysis of Spikes in Atmospheric Imaging Assembly (AIA) Data." Solar Physics 296.12 (2021): 1-21.).
- Shneider, Carl, et al. "A Machine-Learning-Ready Dataset Prepared from the Solar and Heliospheric Observatory Mission." arXiv preprint arXiv:2108.06394 (2021).
- Solar Orbiter Notebooks - Repository holding all the tutorial notebooks for the data analysis day of the Solar Orbiter 8 Workshop.
- STIX Data Center - Spectrometer Telescope for Imaging X-rays (STIX) on Solar Orbiter is a hard X-ray imaging spectrometer covering the energy range from 4 to 150 keV
- AGU Link Collection - Solar physics link collection.
- Astrophysics Data System (ADS) - Digital library portal for researchers in astronomy and physics.
- Carrington rotation date list
- Community Coordinated Modeling Center - Partnership for space science and space weather models.
- Dr Peter R. Young's website - Useful materials for different missions.
- FHNW - Astroinformatics and Heliophysics - Space-related research at the FHNW Institute for Data Science.
- Helionauts - Heliophysics Forum.
- Helioanalytics - Research group and resources for data analytics in heliophysics.
- Heliophysics Data Portal
- Heliophysics Data Environment - Data and Services for the Heliophysics System Observatory.
- LMSAL AIA - Home of SDO's atmospheric imaging assembly (AIA).
- SDO Documentation
- SDO Guide - A comprehensive booklet with information about the mission.
- SDO Mission
- SDO Quick Movie Browser
- Spacewheater.com - News and information about the Sun-Earth environment.
- Spaceweatherlive.com - Real-time auroral and solar activity.
- Space wheater data portal - Tool for visualizing, downloading and co-plotting space wheather data.
- SSCWeb Satellite Situation Center - Locations for most helio spacecraft.
- Sungate - Portal to solar data, events, and search tools.
- Kucuk, Ahmet, Juan M. Banda, and Rafal A. Angryk. "A large-scale solar dynamics observatory image dataset for computer vision applications." Scientific data 4.1 (2017): 1-9. link
- McGranaghan, R., et al. "Machine learning databases used for Journal of Geophysical Research: Space Physics manuscript: New capabilities for prediction of high-latitude ionospheric scintillation: A novel approach with machine learning.". (2018). figshare. Dataset. link
- Ahmadzadeh, Azim, Dustin J. Kempton, and Rafal A. Angryk. "A Curated Image Parameter Data Set from the Solar Dynamics Observatory Mission." The Astrophysical Journal Supplement Series 243.1 (2019): 18. link
- Galvez, Richard, et al. "A machine-learning data set prepared from the NASA solar dynamics observatory mission." The Astrophysical Journal Supplement Series 242.1 (2019): 7. link
- Baek, Ji-Hye, et al. "Solar Event Detection Using Deep-Learning-Based Object Detection Methods." Solar Physics 296.11 (2021): 1-15. link
- Angryk, Rafal A., et al. "Multivariate time series dataset for space weather data analytics." Scientific data 7.1 (2020): 1-13. link code
- Mahajan, Sushant S., et al. "Improved Measurements of the Sun's Meridional Flow and Torsional Oscillation from Correlation Tracking on MDI and HMI Magnetograms." The Astrophysical Journal 917.2 (2021): 100. link data
- Shneider, Carl, et al. "A Machine-Learning-Ready Dataset Prepared from the Solar and Heliospheric Observatory Mission." arXiv preprint arXiv:2108.06394 (2021). link
- Bobra, Monica G., et al. "SMARPs and SHARPs: Two Solar Cycles of Active Region Data." The Astrophysical Journal Supplement Series 256.2 (2021): 26. link code
- Martens, P. C. H., et al. "Computer vision for the solar dynamics observatory (SDO)." Solar Physics 275.1 (2012): 79-113. link
- Banda, Juan M., and Rafal A. Angryk. "Unsupervised learning techniques for detection of regions of interest in Solar Images." 2015 IEEE International Conference on Data Mining Workshop (ICDMW). IEEE, 2015. link
- Schuh, Michael A., Dustin Kempton, and Rafal A. Angryk. "A Region-Based Retrieval System for Heliophysics Imagery." FLAIRS Conference. 2017. link
- Kucuk, Ahmet, Juan M. Banda, and Rafal A. Angryk. "Solar event classification using deep convolutional neural networks." International Conference on Artificial Intelligence and Soft Computing. Springer, Cham, 2017. link
- Illarionov, Egor A., and Andrey G. Tlatov. "Segmentation of coronal holes in solar disc images with a convolutional neural network." Monthly Notices of the Royal Astronomical Society 481.4 (2018): 5014-5021. link
- Kempton, Dustin J., Michael A. Schuh, and Rafal A. Angryk. "Tracking solar phenomena from the sdo." The Astrophysical Journal 869.1 (2018): 54. link
- Armstrong, John A., and Lyndsay Fletcher. "Fast solar image classification using deep learning and its importance for automation in solar physics." Solar Physics 294.6 (2019): 1-23. link
- Gitiaux, Xavier, et al. "Probabilistic Super-Resolution of Solar Magnetograms: Generating Many Explanations and Measuring Uncertainties." arXiv preprint arXiv:1911.01486 (2019). link
- Jungbluth, Anna, et al. "Single-frame super-resolution of solar magnetograms: Investigating physics-based metrics& losses." arXiv preprint arXiv:1911.01490 (2019). link
- Love, Teri, Thomas Neukirch, and Clare E. Parnell. "Analyzing AIA Flare Observations Using Convolutional Neural Networks." Frontiers in Astronomy and Space Sciences 7 (2020): 34. link
- Mackovjak, Šimon, et al. "SCSS-Net: solar corona structures segmentation by deep learning." Monthly Notices of the Royal Astronomical Society 508.3 (2021): 3111-3124. code link
- Broock, Elena García, Tobías Felipe, and A. Asensio Ramos. "Performance of solar far-side active region neural detection." Astronomy & Astrophysics 652 (2021): A132. link
- Innocenti, Maria Elena, et al. "Unsupervised classification of simulated magnetospheric regions." Annales Geophysicae. Vol. 39. No. 5. Copernicus GmbH, 2021. link
- Pesnell, W. Dean, B. J. Thompson, and P. C. Chamberlin. "The solar dynamics observatory (SDO)." The Solar Dynamics Observatory. Springer, New York, NY, 2011. 3-15. link
- Lemen, James R., et al. "The atmospheric imaging assembly (AIA) on the solar dynamics observatory (SDO)." The solar dynamics observatory. Springer, New York, NY, 2011. 17-40. link
- Bobra, Monica G., and Sebastien Couvidat. "Solar flare prediction using SDO/HMI vector magnetic field data with a machine-learning algorithm." The Astrophysical Journal 798.2 (2015): 135. link
- McGregor, Sean, et al. "Flarenet: A deep learning framework for solar phenomena prediction." Neural Information Processing Systems (NIPS) 2017 workshop on Deep Learning for Physical Sciences (DLPS), Long Beach, CA, US. 2017. link
- Nagem, Tarek AM, et al. "Deep learning technology for predicting solar flares from (Geostationary Operational Environmental Satellite) data." (2018) link
- Panos, Brandon, et al. "Identifying typical Mg II flare spectra using machine learning." The Astrophysical Journal 861.1 (2018): 62. link
- Jonas, Eric, et al. "Flare prediction using photospheric and coronal image data." Solar Physics 293.3 (2018): 1-22. link
- McGranaghan, Ryan M., et al. "New capabilities for prediction of high‐latitude ionospheric scintillation: A novel approach with machine learning." Space Weather 16.11 (2018): 1817-1846. link
- Camporeale, Enrico. "The challenge of machine learning in space weather: Nowcasting and forecasting." Space Weather 17.8 (2019): 1166-1207. link
- Chen, Yang, et al. "Identifying solar flare precursors using time series of SDO/HMI Images and SHARP Parameters." Space Weather 17.10 (2019): 1404-1426. link
- Panos, Brandon, and Lucia Kleint. "Real-time flare prediction based on distinctions between flaring and non-flaring active region spectra." The Astrophysical Journal 891.1 (2020): 17. link
- Ivanov, Sergey, et al. "Solar activity classification based on Mg II spectra: towards classification on compressed data." arXiv preprint arXiv:2009.07156 (2020). link
- Wang, Jingjing, et al. "Solar Flare Predictive Features Derived from Polarity Inversion Line Masks in Active Regions Using an Unsupervised Machine Learning Algorithm." The Astrophysical Journal 892.2 (2020): 140. link
- Ahmadzadeh, Azim, et al. "How to Train Your Flare Prediction Model: Revisiting Robust Sampling of Rare Events." The Astrophysical Journal Supplement Series 254.2 (2021): 23. link
- Li, Xuebao, et al. "Predicting solar flares using a novel deep convolutional neural network." The Astrophysical Journal 891.1 (2020): 10. link
- Wang, Xiantong, et al. "Predicting solar flares with machine learning: Investigating solar cycle dependence." The Astrophysical Journal 895.1 (2020): 3. link
- McGranaghan, Ryan M., et al. "Toward a next generation particle precipitation model: Mesoscale prediction through machine learning (a case study and framework for progress)." Space Weather 19.6 (2021): e2020SW002684. link
- Krista, Larisza D., and Matthew Chih. "A DEFT Way to Forecast Solar Flares." The Astrophysical Journal 922.2 (2021): 218. link
- Abduallah, Yasser, et al. "DeepSun: machine-learning-as-a-service for solar flare prediction." Research in Astronomy and Astrophysics 21.7 (2021): 160. link
- Pandey, Chetraj, Rafal A. Angryk, and Berkay Aydin. "Solar Flare Forecasting with Deep Neural Networks using Compressed Full-disk HMI Magnetograms." 2021 IEEE International Conference on Big Data (Big Data). IEEE, 2021. link
- Abed, Ali K., Rami Qahwaji, and Ahmed Abed. "The automated prediction of solar flares from SDO images using deep learning." Advances in Space Research 67.8 (2021): 2544-2557. link
- Nishizuka, Naoto, et al. "Operational solar flare prediction model using Deep Flare Net." Earth, Planets and Space 73.1 (2021): 1-12. link
- Brown, Edward JE, et al. "Attention‐Based Machine Vision Models and Techniques for Solar Wind Speed Forecasting Using Solar EUV Images." Space Weather 20.3 (2022): e2021SW002976. link
- Hu, Andong, et al. "Probabilistic prediction of Dst storms one-day-ahead using Full-Disk SoHO Images" code link
- Huwyler, Cédric, and Martin Melchior. "Using Multiple Instance Learning for Explainable Solar Flare Prediction." arXiv preprint arXiv:2203.13896 (2022). link
- Sun, Zeyu, et al. "Predicting Solar Flares Using CNN and LSTM on Two Solar Cycles of Active Region Data." The Astrophysical Journal 931.2 (2022): 163. link
- Deshmukh, Varad, et al. "Decreasing False-alarm Rates in CNN-based Solar Flare Prediction Using SDO/HMI Data." The Astrophysical Journal Supplement Series 260.1 (2022): 9. link
- Guastavino, Sabrina, et al. "Implementation paradigm for supervised flare forecasting studies: A deep learning application with video data." Astronomy & Astrophysics 662 (2022): A105. link
- Wright, Paul J., et al. "DeepEM: Demonstrating a Deep Learning Approach to DEM Inversion." Zenodo. link
- Guedes dos Santos, L. F., et al. "Multi-Channel Auto-Calibration for the Atmospheric Imaging Assembly instrument with Deep Learning." AGU Fall Meeting Abstracts. Vol. 2020. 2020. link
- Brown, Edward JE, et al. "Learning the solar latent space: sigma-variational autoencoders for multiple channel solar imaging." link
- Jarolim, Robert, et al. "Instrument-To-Instrument translation: Instrumental advances drive restoration of solar observation series via deep learning." (2022). link code
- Aiapy - Package for analyzing calibrated (level 1) EUV imaging data from AIA.
- IRISreader - Python library that allows for efficient browsing through IRIS satellite data in order to simplify machine learning applications.
- Global High-Resolution Hα network
- Helioviewer - Solar and heliospheric image visualization tool. code
- Integrated Solar Database - Solar and heliospheric image visualization tool including image parameters with Extended Spatiotemporal Querying Capabilities.
- integrated Space Weather Analysis (iSWA) system
- SolarMonitor - Provides realtime information about the solar activity. The most recent images of the sun are shown, together with the description of the different NOAA AR of the day and which flares has been associated to them.
- SpaceML - A Machine Learning toolbox and developer community building the next generation AI applications for space science and exploration containing a set of code examples.
- Sunpy - Set of tools for solar data analysis with Python.
- SWPC CME Analysis Tool (SWPC_CAT) - Primary tool being used by NOAA SWPC in measuring key parameters of a Coronal Mass Ejection (CME) code
- The Heliophysics KNOWledge Network - Collection of software and systems for improved information representation in Heliophysics.
- Big data 2020 Conference Talk - Tutorial 6: Data Sources, Tools, and Techniques for Big Data-driven Machine Learning in Heliophysics
- SDO 2021 Science Workshop - Recent science topics targeting SDO data
- SpaceML Youtube Channel - machine learning toolbox and developer community building open science AI applications for space science and exploration.