add functionality to specify temp directory location when creating memory mapped temp arrays #162
Conversation
…mory mapped temp arrays this allows you to also utilize memory mapped temp arrays in multithreaded processes by explicitly passing the directory name without needing to rely on the global TEMP_DIR_NAME variable still being accurate
|
@sophiamaedler We now have quite restrict PR rules for alphaX packages. Please |
@jalew188 Sure :) I had done a git pull before opening this PR, should I still do anything else next time? |
- updated name to temp_dir_abs_path - moved code to check if directory exists to seperate function
move code to check if file path for creating a new empty temp directory is valid to a seperate helper function and update variable name to be more understandable
I think the only thing is to fix this error in this PR: https://github.com/MannLabs/alphabase/actions/runs/9000693343/job/24725402319?pr=162#step:4:246 |
I installed the ruff linter and let it run over the code. I hope that should fix it and should avoid this problem for the future. |
in cases where no abs_path is provided this otherwise throws an error.
rename function to better reflect what the code does
name better reflects what the code does
|
I also noticed that the global definition of TEMP_DIR_NAME was removed and added it back in. This otherwise throws errors in some rare cases. |
In general I think there is a lot of potential here for additional use cases e.g. I have currently been playing around with Dask arrays and found a way to use a tempmmap compatible HDF5 in the backend of a memory mapped Dask array (see here : src/sparcstools/base/daskmmap.py) but I am not sure where the best place to include it would be and if alpha base is the correct location to add something like that. Might be worth a more in depth discussion at some point. |
this allows you to also utilize memory mapped temp arrays in multithreaded processes by explicitly passing the directory name without needing to rely on the global TEMP_DIR_NAME variable still being accurate