LSTM weights not optimised

[timmeinhardt]

The LayerNormLSTMCell modules initialised in the MetaOptimizer class are not properly registered as parameters of the MetaOptimizer model. Appending them to the self.lstms list:

pytorch-meta-optimizer/meta_optimizer.py

Line 27 in 0154d4d

self.lstms.append(LayerNormLSTMCell(hidden_size, hidden_size))

will not add their trainable parameters to the model parameter list in:

pytorch-meta-optimizer/main.py

Line 63 in 0154d4d

optimizer = optim.Adam(meta_optimizer.parameters(), lr=1e-3)

If I am not mistaken, the current version will not train the LSTM weights at all. In general, I would suggest to restructure the initialisation and MetaOptimizer.forward method, but as a quick fix one could replace the entire self.lstms initialization block with this:

self.lstms = nn.Sequential(*[LayerNormLSTMCell(hidden_size, hidden_size)
                                                for _ in range(num_layers)])

[YanwenZhu]

This quick fix worked, thanks! By the way, have you been able to re-implement the experiments which were in the article using MetaOptimizer? For me the final loss of each epoch is really big, which is around 4 for best, and I cannot figure out what to do. Could you please give a little help?

[timmeinhardt]

@YanwenZhu Sorry, but I did not reimplement the original experiments. I applied Learning to learn to an unrelated problem.

[YanwenZhu]

@timmeinhardt Thanks anyway!