ToTensorBoard does not allow to write to non-default file

[Corwinpro]

Feature request

I am not sure if this is a feature request or a bug report. I'd consider both.

There was a fix introduced in #1424 that effectively mutates the global state (bad) implicitly (very bad). In summary, there was an attempt made to allow multiple file writers to write in the same log directory, which works fine for one specific case - everything is written to the same file.

In some cases writing to the same tfevents file is not an option. Internally, tensorflow would generate a new file for each file writer (identified by the process ID and a unique suffix). The tool that allows to analyse the tfevent files (tensorboard) supports multiple files for the same model to exist, so this is a common and supported workflow.

The problem with the ToTensorBoard implementation is that it is not possible to not use the cached writers. Furthermore, it is not possible to know the cache state in advance because it is shared across multiple subclasses. This behaviour is also not documented / tested.

Motivation

Is your feature request related to a problem?

The motivation is: users should be able to control where their data ends up being.

Proposal

Describe the solution you would like

Whilst I understand why the caching solution was implemented, I believe that it also limits the usage of the ToTensorBoard and subclasses. We could implement a session-based caching, such that all file writers refer to the same file within the session, but not otherwise.

What alternatives have you considered?

Are you willing to open a pull request? (We really appreciate contributions!)

Additional context