Improve modelbuilder notebook

[veenstrajelmer]

Notebook was improved in #463

Further improvements:

• now using tpxo which requires large buffer size for plibnd, consider using EOT20 online dataset (not implemented) or tidemodel='GTSMv4.1_opendap' >> buffer can be reduced anyway since we also use nearest interp now
• prevent opendap gebco cutout link in modelbuilder notebook, no API is available, maybe via gebco ddb opendap. Or maybe add coarse dataset to opendap?
• prevent modelbuilder import, expose via dfm_tools directly
• support for waq in modelbuilder example script #504
• Minor modelbuilder improvements #602
• Fill nans in ini sal/tem netcdf #616
• Add initial fields for tracers #618
• CMEMS phyc not available in reanalysis #622
• Improve robustness of CMEMS/CDS authentication #603
• dsd improvements modelbuilder notebook #631
• modelbuilder print modeltree from dimr instead of mdu #725
• simplify cdsapi.Client() call after new cdsapi release #822
• Update to copernicus-marine-client #630
• Improvements in copernicus-marine-client related code #694
• Increase minimal copernicusmarine version #692
• Make conversion_dict an input parameter for the function cmems_nc_to_bc #838
• Add example with ETOPO online dataset #836
• remove os.listdir command after Show_tree output is not printed for old ext files HYDROLIB-core#581 is fixed
• Disruptions and update to new cdsapi #739
• connect to myint cmems reanalyis dataset also #861
• suppress CMEMS and CDS logging in notebook (not in dfmt functions) with import logging; logging.getLogger("copernicus_marine_root_logger").setLevel(level="CRITICAL")
• EDITO: Improve dfm modelbuilder workflow Deltares-research/EditoServices#1
• prevent print of 2023-11-03 14:24:35,649 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/dummy >> quiet argument for Client class?
• consider alternative bathymetry from https://help.marine.copernicus.eu/en/articles/4733481-how-to-access-bathymetry-dataset-used-in-copernicus-marine-model-products
• also support initial values in polygon?
• nans in ini sal/tem are filled per 1d dim (first lat, then lon), consider 2d interpolation
• consider filling source netcdfs for bc files first over lat/lon, instead of only filling bc over depth.
• prevent pydap connection if cmems files are already available, is difficult since we check for available times in products. This is not relevant anymore when we use copernicus-marine-client
• make conversion_dict keys all lowercase and convert all input to lowercase with .lower()? Relevant for e.g. tracerbndNO3. All lowercase is also recognized by FM.
• make conversion_dict a global setting, so a custom conversion dict is automatically used everywhere in the modelbuilding process
• Add function to modelbuilder to generate observations along coastline #575 (or see below example)
• consider creating a function that combines open_dataset_extra, interp_regularnc_to_plipoints, plipointsDataset_to_ForcingModel and maybe more, and remove cmems_nc_to_bc. Is ForcingModel_object.save() required
• conversion of single plifile to multiplifile: Escapes for polyfiles with multiple polylines #492
• test model in windows exe and DeltaShell GUI, not all will work (e.g. lowercase keywords in bc-file are not properly read): Incompatible with Precompiled DFM #710
• add note about importing in GUI with link to Incompatible with Precompiled DFM #710
• adding river discharges (preferrably from a global public datasource) as source/sinks: GRDC has no API and no coastal points. Swedish SMHI EHYPE dataset?
• add dry points (upon grid generation if still necessary)
• add structures, cross-sections, thindams etc (quite complex to to this in a generic way, bit still add support for adding manually)
• Support D-Water Quality input in ExtOldModel HYDROLIB-core#551
• waq: write timfiles for source/sinks (for river discharges in large scale models and WAQ sources), e.g. p:\11209231-003-bes-modellering\hydrodynamica-waq\preprocessing\hydrology_runoff\hydrology_nutrient_to_tim_waq.py (by YC) >> bc files will be implemented in delft3dfm. sourcesinks are also used for river discharges.
• also wastewater: p:\11209231-003-bes-modellering\hydrodynamica-waq\preprocessing\wastewater_20231129\scripts\ (by LB) (is also sourcesinks)
• waq: HYDROLIB-core: Reader of substance file, including comparison to timfile columns. Example model in p:\11209251-sito-veerse-meer\model\runs_2019-2020\Veerse_meer_Model_to_subversion\dflowfm3d_dwaq-veerse_meer-j19_6-v1a_LH. Related issue: If old external forcing file references a pli file, the accompanying .tim/.cmp should also be read and stored HYDROLIB-core#501 (also contains column order of timfiles, so not super important if sourcesinks are forced via bc)
• waq: HYDROLIB-core: ignore or support waq statistics file (is defined in the substance file)
• waq: HYDROLIB-core: support iniwaq* in ext file?
• more WAQ things necessary: contact MW, FD, AM, LvdH

Add sources for e.g. tide models, CMEMS and ERA5:

• UHSLC (public): https://uhslc.soest.hawaii.edu/
• IOC: https://www.ioc-sealevelmonitoring.org/
• GESLA3: https://gesla787883612.wordpress.com/
• FES2014 (not public, commercial paid): https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes/description-fes2014.html
• EOT20 (public): https://essd.copernicus.org/articles/13/3869/2021/
• GTSMv4.1 (public): https://www.deltares.nl/en/expertise/projects/global-modelling-of-tides-and-storm-surges

Add obspoints (or along coastline with #575):

obsfile = os.path.join(dir_output, f"{model_name}_obs.xyn")
obs_x = [-68.2240067, -68.2283020, -68.3852158, -68.1552582]
obs_y = [12.5409269, 12.2554617, 12.0076866, 12.0341187]
obs_n = ['Bonaire01', 'Bonaire02', 'Bonaire03', 'Bonaire04']
xynpoints = [hcdfm.XYNPoint(x=x,y=y,n=n) for x,y,n in zip(obs_x, obs_y, obs_n)]
xynmodel = hcdfm.XYNModel(points=xynpoints)
xynmodel.save(obsfile)

Running the model:

• generate sh file for snellius/h7/amazon, singularity? (bat for windows is handled, as well as generic dimr_config.xml)
• required: singularity, jupyter, maybe file browsing/editing (eg mobaxterm)
• download FM singularity via container harbour instead of download.py (OvD), currently from download portal: https://github.com/Deltares/dfm_tools/pull/488/files
• also make it possible to run the model without modelbuilder, so download+unpack zip with modelfiles?
• binder runs on Kubernetes
• C-scale used docker: https://github.com/c-scale-community/workflow-coastal-hydrowaq/tree/main/fm_model
• singularity on: https://download.deltares.nl/en/delft3dfm-2d3d-beta-singularity (direct download link available)
• add singularity container download #487
• singularity nor docker available on binder per default
• Alternative: singularity also runs on AWS (RdG)
• Alternative: snellius supports interactive jupyter notebook, this should also work on h7 if a node can be reserved for the course
• Alternative: C-SCALE uses https://replay.notebooks.egi.eu/, replay is EGI's implementation of mybinder
• alternatively, EDITO uses jupyter and has connection to HPC
• also possible to use binderhub on different cluster
• if all else fails: zip+download model and run on local machine with bat file

Interactive jupyter slurm:

#!/bin/bash
#SBATCH -t 1:00:00
#SBATCH -N 1

# Make sure the jupyter command is available, either by loading the appropriate modules, sourcing your own virtual environment, etc.
module load 2021
source /home/muiss/miniconda3/etc/profile.d/conda.sh
conda activate /home/muiss/miniconda3/envs/era5-e

# Choose random port and print instructions to connect
PORT=`shuf -i 5000-5999 -n 1`
LOGIN_HOST=${SLURM_SUBMIT_HOST}-pub.snellius.surf.nl
BATCH_HOST=$(hostname)

echo "To connect to the notebook type the following command from your local terminal:"
echo "ssh -J ${USER}@${LOGIN_HOST} ${USER}@${BATCH_HOST} -L ${PORT}:localhost:${PORT}"
echo
echo "After connection is established in your local browser go to the address:"
echo "http://localhost:${PORT}"

jupyter lab --no-browser --port ${PORT}