Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add varlist argument to dfmt.get_Dataset_atdepths() or move layer vars to grid-like acessor #629

Open
veenstrajelmer opened this issue Nov 2, 2023 · 0 comments

Comments

@veenstrajelmer
Copy link
Collaborator

veenstrajelmer commented Nov 2, 2023

With large datasets the dfmt.get_Dataset_atdepths() function is very computationally expensive. Providing a subset of the dataset is possible, but the user should be aware of which layer variables should be passed on. This is inconvenient. In the below example only mesh2d_ucx and mesh2d_ucy are needed by the user, but many variables are required for the depth slicing:

import dfm_tools as dfmt
file_nc = r"p:\11209731-002-nutrient-reduction-ta\runs_OSPAR\B05_waq_withDOM_2016\DFM_OUTPUT_DCSM-FM_0_5nm_waq\DCSM-FM_0_5nm_waq_0000_map.nc"
uds = dfmt.open_partitioned_dataset(file_nc)
uds_sel = uds[[ 'mesh2d_ucx', 'mesh2d_ucy', 
               'mesh2d_layer_sigma', 'mesh2d_layer_z', "mesh2d_layer_sigma_z", "mesh2d_interface_sigma_z", 
               "mesh2d_s1", "mesh2d_bldepth", "mesh2d_sigmazdepth", "mesh2d_interface_z", "mesh2d_interface_sigma",
               ]]
uds_sel_depths = dfmt.get_Dataset_atdepths(uds_sel, depths=[-1])

Desired solution
Add a variables_toslice argument to dfmt.get_Dataset_atdepths(), in this case variables_toslice= ["mesh2d_ucx","mesh2d_ucy"]. If provided, this would overwrite the derived variables_toslice in the function. It is derived based on time/faces/layers dimensions, so when passing a 1D variable, it will also be sliced (eg bedlevel will also get several layers).

Alternative
It might be better to extract the layer data out of the dataset (like the .grid data is also kept separately). This would also likely solve things for d3d4, schism, rst and other formats.

@veenstrajelmer veenstrajelmer changed the title add varlist argument to dfmt.get_Dataset_atdepths() add varlist argument to dfmt.get_Dataset_atdepths() or move layer vars in grid-like acessor May 14, 2024
@veenstrajelmer veenstrajelmer changed the title add varlist argument to dfmt.get_Dataset_atdepths() or move layer vars in grid-like acessor add varlist argument to dfmt.get_Dataset_atdepths() or move layer vars to grid-like acessor May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant