Dask Filling Up Available Disk Space #9836
Replies: 1 comment 1 reply
-
There's not a simple mechanism for explicitly clearing the disk space. However, I would use
I'm curious why things are remaining on disk after your computation is over. Are you holding references to keys that have been spilled to disk? |
Beta Was this translation helpful? Give feedback.
-
Hi,
I'm relatively new to Dask, so apologies in advance if this is a dumb or already answered question, but I haven't been able to find anything online to help.
I have a fairly complex set of Dask operations, including several merges of Dask DataFrames on non-index columns, that result in a pandas DataFrame that fits in main memory no problem. My problem is that this set of operations needs to be repeated several times and when I attempt to do that Dask fills up my available disk space in the C: drive (in AppData/Local). My question is whether there is a simple way to clear all the disk space dask is using in between each iteration of the operations that result in a pandas DataFrame? My current solution is to just restart the computer (this is the only way I can clear the disk space) in between several iterations of the operation so I never fill up the disk. This is obviously a pain, so I'm looking for suggestions on how to fix this!
Thank you!
Beta Was this translation helpful? Give feedback.
All reactions