Executor-specific logging

[vsoch]

I suspect that when more executors are added, it might be useful to have executor-specific logging. I'm going to wait until someone explicitly asks for it, however.

[yarikoptic]

I would not bother too much about it ATM besides giving them dedicated logger/level, e.g. qme.executors.<name> (so you get logger by logging.getLogger(f"qme.executors.{name}") or alike). Then a dedicated formatter etc could be configured for any such specific logger and by default just follow its parent(s) (i.e. "qme.executors" and if that not custom "qme")

[vsoch]

Well, the actual logger is a custom logger I made years ago for singularity - not default python logging. We could switch to default python logging but then it would be an overhaul of my current logger (e.g. bot.<level>()

[yarikoptic]

oh, I saw

$> git grep logging -- qme
qme/app/server.py:import logging
qme/app/server.py:    file_handler = logging.FileHandler(os.path.join(queue.config_dir, "dashboard.log"))
qme/app/server.py:    formatter = logging.Formatter(
qme/app/server.py:    app.logger.setLevel(logging.DEBUG)
qme/logger/message.py:        self.level = get_logging_level()
qme/logger/message.py:def get_logging_level():
qme/logger/message.py:    """get_logging_level will configure a logging to standard out based on
qme/logger/message.py:    # User knows logging levels and set one
qme/logger/progress.py:        # Only show bar in terminals by default (better for piping, logging etc.)

and I thought it was the default one, but indeed seems to be used only in app.server?.

I could only strongly encourage use of standard logger, especially for library projects. It allows to unify logging for downstream applications by e.g. reusing application logger backends etc. We do that in a few spots in datalad - then it becomes quite nice when needed to debug while retaining consistent logging across all used components etc. Should be quite straightforward since likely interface and levels are probably aligned, e.g. seeing

qme/app/views/main.py:    app.logger.debug("Received deletion request for %s" % json.get("taskid"))
qme/app/views/main.py:    app.logger.debug("Received re-run request for %s" % json.get("taskid"))
qme/app/views/main.py:    app.logger.debug("Client connected")
qme/app/views/main.py:    app.logger.debug("Starting Thread")
qme/app/views/main.py:    app.logger.debug("Client disconnected")

probably nothing but replacing .logger object with a standard logger would strictly be needed (although as I had mentioned elsewhere string interpolations could then be delayed and arguments to them passed as arguments to .debug(msg, ...)

[vsoch]

Yeah I have to agree with you here - I’ll want to preserve the bot.table function and just put it somewhere else.