Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Appending data to a DataArray in Dfs2 file #654

Closed
AL89 opened this issue Feb 20, 2024 · 2 comments · May be fixed by #691
Closed

Appending data to a DataArray in Dfs2 file #654

AL89 opened this issue Feb 20, 2024 · 2 comments · May be fixed by #691

Comments

@AL89
Copy link

AL89 commented Feb 20, 2024

Hi there

I am in a bit of a struggle. And I hope my problem has a solution.

It's all about to avoid memory errors, like: MemoryError: Unable to allocate 1.52 GIB for an array with shape (41760,75,65) and data type float64.
I am seeking a solution where I can append my data stored in my memory into an existing Dfs2 file, without loading the existing data into memory. Like a "load and dump" solution.

The example above with the MemoryError, the first "dimension" is the time as you might know, which is 41760 timesteps with timestep of 60 seconds. This means that it tried to load 29 days before it "crashed"... and I would like to have a year.

When loading Dfs2 from the mikeio, I am able to see some details about the file in-memory. Is it possible to store data in the existing DataArray (not Dataset) in this mode?

from mikeio import Dfs2
import numpy as np

dfs2 = Dfs2('data.dfs2')   # Assuming this is the existing Dfs2 file.
test_data = np.zeros(dfs2.shape)
# ds = dfs2.read()            # I would like to avoid this...
dfs2.write(to_existing=DataArray,data=test_data)    # Of course, 'to_existing', is a made up argument, but something like this.

Is there any solution to my memory problem?

Thanks in advance!

@ecomodeller
Copy link
Member

I think there are several solutions or parts:

  1. Use mikeio.generic.concat to concatenate files
  2. Use mikecore directly and skip MIKE IO, then you can append to an existing Dfs2 file.
  3. Use float32 instead of float64
  4. Rent a cloud computer with more memory

@AL89
Copy link
Author

AL89 commented Feb 21, 2024

Hi @ecomodeller
I tried the first solution and it worked like a charm! Thank you!

@ecomodeller ecomodeller linked a pull request May 10, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants