Dreamer implementation

[GaspardCulis]

Hi,

Do you plan on adding a clean Dreamer implementation ? From what I've read it's pretty state of the art stuff.

Cheers

[Howuhh]

To be honest, I don't think that implementing dreamer in one file and implementing it clean is even possible, given the amount of different hacks and overall difficulty of the whole architectures involved...

[vwxyzjn]

Yeah it seems to require a significant amount of effort and compute.

[pseudo-rnd-thoughts]

It might be possible as there is the official source code available at https://github.com/danijar/dreamerv3
However looking at it, it is a mess of functions with almost no documentation for what is happening

[Howuhh]

Actually, this is the reason why such reimplementation would be extremely useful for the practitioners and researchers. However, this also will be very time consuming as @vwxyzjn pointed out.

[vwxyzjn]

We actually looked into it as part of our work in https://neurips.cc/virtual/2023/poster/72315, but it was just very challenging. Its experiments look extremely expensive. We were told "for each Atari game, it took 16 days on 1xV100 to train the agent." This results in a median human normalized score (HNS) of 302% in 55 Atari games.

In Cleanba IMPALA we can achieve 160% median HNS in ~160 minutes with 1xA100. While 160% < 302%, the compute time is 160 / (16*24*60) = 0.6% of the runtime of Dreamer-V3. It's obviously not a direct comparison (different hardware, different size neural networks (e.g., our IMPALA net has about 1M parameters Dreamer v3 uses XL which has 200M parameters), also dreamer-v3 is not really for Atari; it's more for MineCraft, but nevertheless this makes reproduction of it challenging and I feel people would prob better off with Cleanba PPO / IMPALA.

[pseudo-rnd-thoughts]

Fascinating paper, this is a feeling I have got a couple of times recently looking at MEME (Agent57 upgrade) and other new training algorithms that it feels like there are tonnes of heuristical improvements in different papers which is difficult to assess if these are special to just this algorithm or more general improvements to RL.
I'm expecting a rainbow like paper to clean this all up but that is probably not possible anymore with the number of heuristics, testing a number of algorithms and environments etc.
Shame that most of the improvements had limited improvements to PPO

[Howuhh]

I think sooner or later Dreamer will be made faster. Right now it seems that training time is inflated due to its inefficiency, as it needs to reconstruct states (which is very expensive for images) (can be solved with some latent consistency losses) and from the sequential nature of RNNs used (can be solved with S5). For now it is not worth it imho.