01_netcdf_format.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To demonstrate our recommandations on data preprocessing and quality control, we decided to use 1-min solar raidation measurements from the BSRN network (https://bsrn.awi.de/). This dataset encompasses among other variables high quality measurements of the three components of the solar radiation for a set of stations worldwide. For the present exercise, we have created netcdf data container for each station including the whole history of the three component measurements (other parameters have been left). Each netcdf contains the following fields:\n",
    "\n",
    "\n",
    "<table>\n",
    "<tbody>\n",
    "<tr><td><b>field-name   </td><td>time                     </td><td>time_bnds         </td><td>Lat          </td><td>Lon         </td><td>elevation                     </td><td>BNI                                              </td><td>DHI                                       </td><td>GHI                                       </td></tr>\n",
    "<tr><td><b>long_name    </td><td>Time of the end of period</td><td>Integration period</td><td>Latitude     </td><td>Longitude   </td><td>Elevation above mean sea level</td><td>Beam Normal Irradiance                           </td><td>Diffuse Horizontal Irradiance             </td><td>Global Horizontal Irradiance              </td></tr>\n",
    "<tr><td><b>standard_name</td><td>Time                     </td><td>-                 </td><td>Latitude     </td><td>Longitude   </td><td>Elevation above mean sea level</td><td>surface_direct_downwelling_ shortwave_flux_in_air</td><td>diffuse_downwelling_ shortwave_flux_in_air</td><td>surface_downwelling_ shortwave_flux_in_air</td></tr>\n",
    "<tr><td><b>units        </td><td>days since 1970-01-01    </td><td>-                 </td><td>degrees_north</td><td>degrees_east</td><td>m                             </td><td>W/m^2                                            </td><td>W/m^2                                     </td><td>W/m^2                                     </td></tr>\n",
    "<tr><td><b>dimensions   </td><td>(T)                      </td><td>(T)               </td><td>(Y)          </td><td>(X)         </td><td>(Z)                           </td><td>(T,Y,X,Z)                                        </td><td>(T,Y,X,Z)                                 </td><td>(T,Y,X,Z)                                 </td></tr>\n",
    "<tr><td><b>missing_value</td><td>n.a.                     </td><td>n.a.              </td><td>n.a.         </td><td>n.a.        </td><td>n.a.                          </td><td>NaN                                              </td><td>NaN                                       </td><td>NaN                                       </td></tr>\n",
    "<tr><td><b>instrument   </td><td>n.a.                     </td><td>n.a.              </td><td>n.a.         </td><td>n.a.        </td><td>n.a.                          </td><td>instrument3                                      </td><td>instrument2                               </td><td>instrument1                               </td></tr>\n",
    "<tr><td><b>comment      </td><td>-                         </td><td>-                  </td><td>-             </td><td>-            </td><td>-                              </td><td>BNI set to 0 for SZA &gt;= 97 deg                   </td><td>DHI set to 0 for SZA &gt;= 97 deg            </td><td>GHI set to 0 for SZA &gt;= 97 deg            </td></tr>\n",
    "</tbody>\n",
    "</table>\n",
    "\n",
    "The field information of the netcdf have been chose in order to match the climate and forecast conventions so that the netcdf is interoperable. Further information on the climate and forecast conventions can be found on:\n",
    "https://www.unidata.ucar.edu/software/netcdf/conventions.html\n",
    "\n",
    "In addition, important metadata are encoded in each netcdf to allows tracking most important information on the measurements. The most relevant metadata contained in the netcdf are listed in the table below:\n",
    "\n",
    "\n",
    "<table>\n",
    "<tbody>\n",
    "<tr><td><b>field-name  </td><td>instrument1            </td><td>instrument1_calibration</td><td>instrument2                                </td><td>instrument2_calibration</td><td>instrument3             </td><td>instrument3_calibration</td></tr>\n",
    "<tr><td><b>long_name   </td><td>Thermopile pyranometer </td><td>n.a.                   </td><td>Thermopile pyranometer with a shadding ball</td><td>n.a.                   </td><td>Thermopile pyrheliometer</td><td>n.a.                   </td></tr>\n",
    "<tr><td><b>text        </td><td>-                      </td><td>-                      </td><td>-                                          </td><td>-                      </td><td>-                       </td><td>-                      </td></tr>\n",
    "<tr><td><b>calibration </td><td>instrument1_calibration</td><td>n.a.                   </td><td>instrument2_calibration                    </td><td>n.a.                   </td><td>instrument3_calibration </td><td>n.a.                   </td></tr>\n",
    "<tr><td><b>precision   </td><td>                       </td><td>n.a.                   </td><td>                                           </td><td>n.a.                   </td><td>                        </td><td>n.a.                   </td></tr>\n",
    "<tr><td><b>zenithDegr  </td><td>                       </td><td>n.a.                   </td><td>                                           </td><td>n.a.                   </td><td>                        </td><td>n.a.                   </td></tr>\n",
    "<tr><td><b>azimuthDegr </td><td>                       </td><td>n.a.                   </td><td>                                           </td><td>n.a.                   </td><td>                        </td><td>n.a.                   </td></tr>\n",
    "<tr><td><b>date        </td><td>n.a.                   </td><td>                       </td><td>n.a.                                       </td><td>                       </td><td>n.a.                    </td><td>                       </td></tr>\n",
    "<tr><td><b>responsivity</td><td>n.a.                   </td><td>                       </td><td>n.a.                                       </td><td>                       </td><td>n.a.                    </td><td>                       </td></tr>\n",
    "<tr><td><b>unit        </td><td>n.a.                   </td><td>                       </td><td>n.a.                                       </td><td>                       </td><td>n.a.                    </td><td>                       </td></tr>\n",
    "<tr><td><b>comment     </td><td>                       </td><td>                       </td><td>                                           </td><td>                       </td><td>                        </td><td>                       </td></tr>\n",
    "</tbody>\n",
    "</table>\n",
    "\n",
    "\n",
    "The netcdf files have then been uploaded on  Thredds data server (TDS) which allows their findability and accessabillity. Thereby, our data comply with the FAIR principles for scientific data management. In particular, the data can be easily manipulated and downloaded using the openDAP functionality of the Thredds server. This option is used below to load the data.\n",
    "\n",
    "\n",
    "In our example detailed below, we load measurements from the Payern BSRN station in Switzerland and will conduct the quality control for the three first years of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}