Caching 'static' compute results

[Peder2911]

Some transforms require computations that can be cached to increase performance and decrease the number of database requests. An idea would be to cache these results in the directory. We'd need to handle invalidation, since the computations are not perpetually valid.

[Peder2911]

For starters, I'd write a class like this:

class CompCache():
   _ROOT_DIR = "~/.views"

   @property
   def _root_dir(self):
      return os.path.expanduser(self._ROOT_DIR)

   def __init__(self, cache_name: str = "transform_library"):
      self._name = cache_name
      self._ensure_folder_struct()

   def _ensure_folder_struct(self):
      try:
         os.makedirs(self._path(""))
      except FileExistsError:
         pass

   def _path(self, name):
      os.path.join(self._root_dir, self._name, name)

   def set(self, name, object):
      with open(self._path(name), "wb") as f:
         pickle.dump(object, f)

   def get(self, name):
      with open(self._path(name), "rb") as f:
         return pickle.load(f)

This does not do any sort of invalidation / timestamping.

[mihaicroicu]

Use diskcache instead! Much better, handles everything you need out of the box, has a FIFO queue for storage size handling, invalidates everything at a given time, and serializes into pickles. And serializes at both object and function level using decorators. BTW, os.makedirs has an exist_ok flag so you don't need that custom method.

[Peder2911]

Good idea!

[Peder2911]

Although it seems a bit overkill maybe? If we have time to learn it, I'm sure it'd be worth the investment, but this is a pretty simple file-caching case, so we don't need a lot of bells and whistles.