Add NNCG to optimizers submodule

[pratikrathore8]

No description provided.

[lululxvi]

Format the code using black https://github.com/psf/black

[pratikrathore8]

I've formatted the code with black in the latest commit!

[lululxvi]

If this only supports pytorch, then it should be in https://github.com/lululxvi/deepxde/tree/master/deepxde/optimizers/pytorch

[pratikrathore8]

Moved to the right folder!

[pratikrathore8]

Made the formatting changes. I noticed there are other lines in the code with > 88 chars per line. Would you like for me to fix those?

[lululxvi]

Made the formatting changes. I noticed there are other lines in the code with > 88 chars per line. Would you like for me to fix those?

Yes.

Could you also fix the Codacy issues?

[pratikrathore8]

The formatting has been fixed.

There are still some Codacy issues, but these are mainly due to "too many local variables" and the __init__ function in the optimizer, which is just following PyTorch conventions.

What do we need do next to further integrate NNCG into the codebase?

[lululxvi]

Need to add the NNCG to https://github.com/lululxvi/deepxde/blob/master/deepxde/optimizers/pytorch/optimizers.py

[pratikrathore8]

I've added NNCG to optimizers.py and NNCG_options to config.py

[pratikrathore8]

Fixed!

[pratikrathore8]

Fixed!

[lululxvi]

Have you tried using NNCG in some demo examples?

[pratikrathore8]

A quick update:

I've started thinking about how to make a demo example that uses NNCG. The plan would be to modify some of the existing examples (e.g. Burgers.py) and add on NNCG after Adam and L-BFGS.

I'll have to make some modifications to model.py to full integrate NNCG into the training pipeline. I'm aiming to have an example fully working by the end of next week!

[pratikrathore8]

I've added a demo Burgers_NNCG.py in examples that uses NNCG. NNCG consistently lowers the loss after Adam+L-BFGS, although there is variance in the amount of improvement (which is probably due to not setting the random seed). Please let me know if you see the improvements on your end too!

To fully integrate NNCG into the codebase, I added a function called _train_pytorch_nncg to model.py. I've left comments in _train_pytorch_nncg to describe my thought process.

Let me know what changes need to be made! I'm also happy to add more demos if you have suggestions.

[pratikrathore8]

@lululxvi Any updates on the pull request?