Support for UniformScaling

[wesselb]

UniformScaling is currently not supported:

julia> Leaf(Tape(), 0) + UniformScaling(1)
ERROR: MethodError: no method matching +(::Nabla.Leaf{Int64}, ::UniformScaling{Int64})

[willtebbutt]

We need to be careful here. Apparently we're suppose to use

5 + UniformScaling(1).λ

as

5 + UniformScaling(1)

is deprecated. But yes, we definitely should support.

[iamed2]

2I is another way of spelling UniformScaling(2), by the way

[willtebbutt]

As it turns out, this is actually quite a troublesome implementation. Consider the reverse-mode sensitivity of

Y = 5 * UniformScaling(x)

w.r.t. x, where x is a scalar. If the reverse-mode sensitivity of Y is itself a UniformScaling, then the answer is undefined because we have not provided any information regarding the size of the matrices that any of the UniformScalings are meant to represent. If, only the other hand, Ȳ has a particular size, say it's N x N, then the solution is well defined (it's 5 * sum(diag(Ȳ))).

Phrased differently, the reverse-mode sensitivity of x is x̄ = 5 * sum(diag(Ȳ)). If Ȳ has a constant diagonal, say β, then this becomes x̄ = 5 N β where Y has size N x N. If Ȳ is itself represented by a UniformScaling, then we don't know N.

There are other problems than the one I've presented here, but they pretty much all seem to stem from UniformScalings not coming with any size information; it's the things with which they interact that give them some notion of size.

Am I thinking about this the right way, or have I missed something obvious?