Filename from environment variable in save

[dojeda]

On three separate occasions, I have needed to set the exact name of a filename in a Save node, instead of the auto-generated one. In particular, I needed to have an HDF5 and a log file with matching names. Since the log filename is already configurable through an environment variable, I kept this approach. For this use case, I created a workaround node that does as follows:

import os

from timeflux.nodes.hdf5 import Save as TimefluxSave


class Save(TimefluxSave):

    def __init__(self, *, path_env=None, filename_env=None, **kwargs):
        if path_env is not None:
            path = os.getenv(path_env, None)
            if path is not None:
                kwargs.setdefault('path', path)

        if filename_env is not None:
            filename = os.getenv(filename_env, None)
            kwargs.setdefault('filename', filename)

        super().__init__(**kwargs)

Will you be interested in creating an improvement on the timeflux.nodes.hdf5.Save node to add this behavior?

[mesca]

Using environment variables is definitely the right approach, and I want to expand on that idea. Instead of implementing the behavior you propose at the node level, why would not we apply this at the graph level? We could have strings like ${MY_VAR} in params, and have them parsed in the manager or the worker. We could also define a bunch of generic env vars like ${DATETIME}, ${ROOT}, etc.