Register weights as a non-persistent buffer of SinusoidalPositionalEmbedding

[MaigoAkisame]

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  Discussed this in a GitHub issue of the pytorch repo.

• Did you read the contributor guideline?

• Did you make sure to update the docs?
  Not applicable

• Did you write any new necessary tests?
  No. But I've tested deeplearning/projects/fairseq-py:test_cpu in Meta's fbcode repo, and this diff does not introduce any new test failures.

What does this PR do?

The module SinusoidalPositionalEmbedding has the problem that its weights attribute is not moved to CPU or CUDA when the module is moved.

Registering weights as a buffer solves the problem.
This also eliminates the need for the buffer _float_tensor, which is used to keep track of whether the module is on CPU or CUDA.

Making weights a non-persistent buffer means it won't be saved to or loaded from a state_dict.

With the changes in this diff, the state_dict of a SinusoidalPositionalEmbedding module should contain neither weights or _float_tensor.
This diff ignores them by overriding the _load_from_state_dict method of the SinusoidalPositionalEmbedding module, instead of duplicating the code in many upgrade_state_dict functions.

TO DISCUSS: Is it OK for me to override _load_from_state_dict? It's a private function, but I see people have overriden it in many places, including in fairseq:
https://github.com/search?q=super()._load_from_state_dict&type=code
https://github.com/search?q=repo%3Afacebookresearch%2Ffairseq%20super()._load_from_state_dict&type=code

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[dianaml0]

How come this is removed?

[MaigoAkisame]

The _float_tensor buffer was just a dummy tensor to track whether the SinusoidalPositionalEmbedding object is on CPU or CUDA. Now the weights buffer can be moved between CPU and CUDA, we no longer need _float_tensor.

[dianaml0]

Does this keep backwards compatability?

[MaigoAkisame]

Does this keep backwards compatability?

Yes. The buffer weights is created upon construction of a SinusoidalPositionalEmbedding object; it doesn't need to be loaded from a state_dict.
With the changes in this PR, no matter whether a state_dict contains the keys weights and _float_tensor or not, these keys will be ignored.

[MaigoAkisame]

BTW In the discussion on the pytorch issue, they've said it's OK to override the _load_from_state_dict method.

[dianaml0]

LGTM!