[onert] Apply BackPropAccumulator layer

[ragmani]

This commit applies BackPropAccumulator layer to train backend.

• Add registering and planing disposable tensors for back-propagation
• Make layers use disposable tensors instread of original back-prop tensors
• Apply BackPropAccumulator layer to each BackPropTensor

ONE-DCO-1.0-Signed-off-by: ragmani ragmani0216@gmail.com

[ragmani]

After this change, training of the branching graph becomes possible.

$ ./Product/x86_64-linux.release/out/bin/onert_train mnist_branched.circle --load_expected:raw out/train.output.1000.bin --load_input:raw out/train.input.1000.bin --loss 1 --loss_reduction_type 1 --optimizer 1 --learn
ing_rate 0.001 --batch_size 1
Model Expected Filename out/train.output.1000.bin
Model Input Filename out/train.input.1000.bin
Model Filename mnist_branched.circle
== training parameter ==
- learning_rate   = 0.001
- batch_size      = 1
- loss_info       = {loss = mean squared error, reduction = sum over batch size}
- optimizer       = sgd
========================
Epoch 1/5 - time: 0.196ms/step - loss: [0] 0.0340
Epoch 2/5 - time: 0.175ms/step - loss: [0] 0.0316
Epoch 3/5 - time: 0.186ms/step - loss: [0] 0.0304
Epoch 4/5 - time: 0.197ms/step - loss: [0] 0.0296
Epoch 5/5 - time: 0.180ms/step - loss: [0] 0.0289
===================================
MODEL_LOAD   takes 0.3950 ms
PREPARE      takes 2.3390 ms
EXECUTE      takes 950.4810 ms
- Epoch 1      takes 195.6460 ms
- Epoch 2      takes 175.2290 ms
- Epoch 3      takes 185.9180 ms
- Epoch 4      takes 196.5610 ms
- Epoch 5      takes 179.5790 ms
===================================

[jyoungyun]

LGTM

[zetwhite]

Could you give me an example of BackwardTensor?

In my understanding,

• DisposableBackwardTensor: Tensor for backpropagation. Once propagation is done, It can be freed.
• BackwardTensor: Used for backpropagation but can not be freed..? I couldn't think of a concrete example of this tensor.

[ragmani]

This code just registers information of tensors that are only used in backwarding. It does not directly affect memory planning of any tensors. Memory planning of DisposableBackPropTensors will be planed in planDisposableBackPropTensors() below. And other tensors for backwarding do not have memory planning yet because they don't have def/use information.
If you are curious about how memory planning is possible with the function below, I will be happy to provide additional explanation online or offline.

[zetwhite]

LGTM 👍

[zetwhite]

I have another little question.

Is it possible to replace tgraph.btopolSorrtOperations() with an operation vector(backward_order) that applied truncatedBackwardOrder?

ONE/runtime/onert/core/src/compiler/ExecutorFactory.cc

Lines 750 to 753 in 63e81d0

	// linearize for backwarding
	auto backward_order = lowered_graph->trainable_graph().btopolSortOperations();
	// get rid of all nodes not reachable from a node with trainable parameters
	backward_order = lowered_graph->trainable_graph().truncateBackwardOrder(backward_order);

I thought that if backward is not going to work, there is no need to allocate tensor related to that.

[zetwhite]

BTW, This is a bit out of context. So, please proceed without considering this question.

[ragmani]

Thanks for your information. I missed out truncateBackwardOrder.

[ragmani]

@Aeren1564 Are you going to apply truncateBackwardOrder to all where btopolSortOperations() is being used? If so, Could you apply it to here as well?