possible addition: fwd diff of R^N -> R using multiple passes

[dsyme]

In investigating simultaneous hyper-parameter optimization of VAE learningRate and beta1 I found that using reverse-on-reverse mode optimization (via SGD outer and Adam inner) led to out-of-memory

I switched SGD to "fwd diff using multiple passes" - in this case two passes, using this code to replace dsharp.fg = dsharp.fdiff in SGD:

    static member fwdGrad f (x:Tensor) =
        if x.dim > 1 then failwithf "f must be a function of a scalar or vector, encountered f:%A->?" x.shape 
        if x.dim = 0 then dsharp.fdiff f x
        else 
            let fxs, ds = Array.init x.nelement (fun i -> dsharp.evalForwardDiff f x (x.onehotLike(x.nelement, i))) |> Array.unzip
            fxs.[0], dsharp.stack ds

I think the implementation is correct (?)

Should this be in the differentiation API since fwd has better memory properties than reverse? Should Adam and SGD take an optional parameter to force the use of forward-mode?

We could also investigate memory usage propreties for this example of reverse-on-reverse. Maybe I was doing somethnig wrong.

As an aside, just to mention again I find the abbreviations like dsharp.fg unhelpful. If they are standard abbreviations that's fine, but there is so much confusion already over "f", "g", etc. that I have no really idea of the system here.