Forward unification 1: add output type family, HasForward instances #477
Conversation
mcwitt
force-pushed
the
forward-unification-1
branch
from
bcf0381
to
47ebd27
Compare
| @@ -34,9 +35,9 @@ dropoutForward :: | |||
| IO (Tensor device dtype shape) | |||
| dropoutForward Dropout {..} dropoutTrain = dropout dropoutProb dropoutTrain | |||
|
|
|||
| instance HasForward Dropout (Tensor device dtype shape) (Tensor device dtype shape) where | |||
| instance HasForward Dropout (Tensor device dtype shape) where | |||
| type Output Dropout (Tensor device dtype shape) = Tensor device dtype shape | |||
| forward dropout input = unsafePerformIO $ dropoutForward dropout False input | |||
There was a problem hiding this comment.
alright, I guess here we are blocked because there isn't a dropout implementation yet that supports generators
There was a problem hiding this comment.
Yes, makes sense. My initial goal was just to get everything to compile, but makes sense that this is not very useful until we can augment the output type to make it stochastic.
|
@mcwitt This looks great! |
| Output | ||
| (TransformerLM numAttnLayers numHeads ffnDim paddingIdx numEmbeds embedDim dtype device) | ||
| (Tensor device 'D.Int64 '[batchSize, seqLen]) = | ||
| Tensor device dtype '[batchSize, seqLen, numEmbeds] | ||
| forward model input = unsafePerformIO $ transformerLM model False input |
There was a problem hiding this comment.
same here. I hope we can get a dropout soon that takes a generator as argument. otherwise, we can't train these models with dropout.
There was a problem hiding this comment.
Yeah, too bad. When I find some time I'm hoping to try to reproduce the test failure in Junji's PR and see if I can make any progress..
|
hey, do you want to revisit this for |
@tscholak yep, I'm pretty excited to try this out! |
Breaking up #459 into a series of more manageable chunks.
In this PR the goals are to:
Update
HasForward, replacing output type parameterbwith a type family and removingfowardStoch. Stochastic vs. deterministic will now be determined by the associated type family for each instance, with types of the formGenerator -> (a, Generator)stochastic.Add
HasForwardProduct,HasForwardSuminstances for products and sums. Eventually (not in this PR), we can dispatch toforwardProductandforwardSumvia a generic implementation offorward.Get everything to compile! This means updating all existing
HasForwardinstances.HasForwardinstance progress