WIP feat:Init commit for rust backend #1180

Aisuko · 2023-10-17T02:37:01Z

Description

This PR relates to #939

Notes for Reviewers

Signed commits

Yes, I signed my commits.

Signed-off-by: GitHub <noreply@github.com>

mudler · 2023-10-17T16:36:36Z

cc @lu-zero

.gitignore

backend/rust/Makefile

backend/rust/src/main.rs

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

Signed-off-by: GitHub <noreply@github.com>

Signed-off-by: Aisuko <urakiny@gmail.com>

backend/rust/bunker/build.rs

backend/rust/burn/src/main.rs

Signed-off-by: Aisuko <urakiny@gmail.com>

backend/rust/bunker/src/service.rs

backend/rust/bunker/src/lib.rs

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

backend/rust/burn/src/main.rs

Signed-off-by: Aisuko <urakiny@gmail.com>

lu-zero · 2023-10-31T08:48:30Z

it seems to look for libtorch and fails to find it. if you use the ndarray backend does it work?

Aisuko · 2023-10-31T23:37:39Z

it seems to look for libtorch and fails to find it. if you use the ndarray backend does it work?

Will try it and give a feedback

Update

ndarary backend can be used to debug in IDE. And the torch backend has some issues on Mac M1. Here I am trying to set up LIBTORCH_USE_PYTORCH=1 as env with the conda env which is installed PyTorch. However, it is still hit other issues on M1 environment. So, I'm going to use ndarray to help me debug the conversion part code.

lu-zero · 2023-11-01T08:37:49Z

On the M1 probably the wgpu backend is the nicest to use, but ndarray is the one that does not depend on the host system.

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko · 2023-11-01T11:09:56Z

On the M1 probably the wgpu backend is the nicest to use, but ndarray is the one that does not depend on the host system.

Thanks a lot. I have made some change here. I have been migrated the code which is included Llama2 to fork repo, and I am working on the a more simpler model. Here are some reasons:

A simpler model can be more effecient to debug than Llama2, less parameters, and less memory used. (Only load half of Llama2 parameters to tensor can cost at least 13min in my local env now)
We can move faster on this PR. It is good for us to refractor the code, project structure and abstract some common traits.
Easy for code reviewing
Easy for adding some test cases(CI).

Here I hit an issue on reshaping of the Tensor. So, we can try to implement a simple one instead of getting stuck on the Llama2.

backend/rust/models/src/lib.rs

Signed-off-by: Aisuko <urakiny@gmail.com>

lu-zero · 2023-11-16T06:34:15Z

backend/rust/models/src/whisper/utils.rs

+        // And now the nonlinear scale
+        let min_log_hz = 1000.0; // beginning of log region (Hz)
+        let min_log_mel = (min_log_hz - f_min) / f_sp;
+        let logstep = (6.4f64).ln() / 27.0; // step size for log region


those constants are repeated, being always f64 you can just keep them as consts

thank you, will do.

Signed-off-by: Aisuko <urakiny@gmail.com>

backend/rust/backend/src/main.rs

Signed-off-by: Aisuko <urakiny@gmail.com>

netlify · 2023-11-23T01:15:35Z

❌ Deploy Preview for localai failed.

Name	Link
🔨 Latest commit	`c990112`
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/655ea7b3d02aec0008ca4cdf

Aisuko · 2023-11-23T01:22:15Z

backend/rust/models/src/llama/llama.rs

+
+        let tensor3=tensor2.transpose();
+
+        let tensor41=tensor3.repeat(2, 2);


@lu-zero Here, I am going to use wgpu backend instead of tch. However, I the repeat function here only support 2 dimensions tensor, (Can only repeat dimension with dim=1) https://github.com/Tracel-AI/burn/blob/b86bc5876149bd73bc59cb5197fd3ee8b92509d4/burn-tensor/src/tensor/ops/tensor.rs#L222C7-L222C7.

I have been tried several solutions, like use swap_dims and flattern these internal function of Tensor, but here hard to say it is correct and also causes other issues. Is there a better example for this?

asking upstream probably it is the best route (sorry for the belated reply, I got very busy and the message got lost in the mailbox)

Aisuko marked this pull request as draft October 17, 2023 02:38

Aisuko self-assigned this Oct 17, 2023

Aisuko force-pushed the feat/rust_grpc branch 2 times, most recently from aacaf4e to afcd7bd Compare October 17, 2023 04:53

Init commit for rust backend

ef3fe9a

Signed-off-by: GitHub <noreply@github.com>

Aisuko force-pushed the feat/rust_grpc branch from afcd7bd to ef3fe9a Compare October 17, 2023 04:57

Aisuko added the new-backend label Oct 17, 2023

lu-zero reviewed Oct 17, 2023

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/Makefile Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/src/main.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 17, 2023

View reviewed changes

backend/rust/src/main.rs Outdated Show resolved Hide resolved

Aisuko and others added 3 commits October 18, 2023 10:47

Update backend/rust/Makefile

029a71f

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

Add tracing

5c67aa6

Signed-off-by: GitHub <noreply@github.com>

Add workspace

1806dd7

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko requested a review from lu-zero October 18, 2023 08:00

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/bunker/build.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 18, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

Aisuko requested a review from lu-zero October 19, 2023 04:51

Replace the generated file to the generated folder

61bd269

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko force-pushed the feat/rust_grpc branch from 9b74e4e to 61bd269 Compare October 19, 2023 08:43

lu-zero reviewed Oct 19, 2023

View reviewed changes

backend/rust/bunker/src/service.rs Outdated Show resolved Hide resolved

lu-zero reviewed Oct 19, 2023

View reviewed changes

backend/rust/bunker/src/lib.rs Outdated Show resolved Hide resolved

Update backend/rust/bunker/src/lib.rs

b92677b

Co-authored-by: Luca Barbato <luca.barbato@gmail.com> Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko commented Oct 20, 2023

View reviewed changes

backend/rust/burn/src/main.rs Outdated Show resolved Hide resolved

Remove services.rs

a2bb86f

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko force-pushed the feat/rust_grpc branch from ef8a86b to a2bb86f Compare October 20, 2023 00:36

Aisuko requested a review from lu-zero October 20, 2023 01:09

Add test health in Makefile

bc6c1fc

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko added 2 commits November 1, 2023 20:12

Add new model

c0dadcc

Signed-off-by: Aisuko <urakiny@gmail.com>

Implement a new simple model

fb67c91

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko force-pushed the feat/rust_grpc branch from 4c7f5ca to fb67c91 Compare November 1, 2023 10:55

Aisuko force-pushed the feat/rust_grpc branch 3 times, most recently from d9f1f7d to da3a0d8 Compare November 3, 2023 11:50

Aisuko commented Nov 3, 2023

View reviewed changes

backend/rust/models/src/lib.rs Show resolved Hide resolved

Implement MNIST model and inference

1d2fd99

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko force-pushed the feat/rust_grpc branch 4 times, most recently from cb216fa to ed95d9c Compare November 4, 2023 02:26

Add check memory feature

660cc49

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko force-pushed the feat/rust_grpc branch from ed95d9c to 660cc49 Compare November 4, 2023 02:43

Aisuko mentioned this pull request Nov 5, 2023

[EPIC] Model support dashboard (v2) #1126

Open

88 tasks

lu-zero reviewed Nov 16, 2023

View reviewed changes

Aisuko force-pushed the feat/rust_grpc branch 3 times, most recently from a6ff963 to b91b79c Compare November 18, 2023 01:35

Trying to call mnist model in main

d62c701

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko force-pushed the feat/rust_grpc branch from b91b79c to d62c701 Compare November 18, 2023 07:29

Add test case for load model and import getusage

b210203

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko commented Nov 19, 2023

View reviewed changes

backend/rust/backend/src/main.rs Show resolved Hide resolved

Add llama for test

c990112

Signed-off-by: Aisuko <urakiny@gmail.com>

Aisuko commented Nov 23, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP feat:Init commit for rust backend #1180

WIP feat:Init commit for rust backend #1180

Aisuko commented Oct 17, 2023 •

edited

mudler commented Oct 17, 2023

lu-zero commented Oct 31, 2023

Aisuko commented Oct 31, 2023 •

edited

lu-zero commented Nov 1, 2023

Aisuko commented Nov 1, 2023

lu-zero Nov 16, 2023

Aisuko Nov 17, 2023

netlify bot commented Nov 23, 2023 •

edited

Aisuko Nov 23, 2023

lu-zero Dec 21, 2023


		let tensor3=tensor2.transpose();

		let tensor41=tensor3.repeat(2, 2);

WIP feat:Init commit for rust backend #1180

Are you sure you want to change the base?

WIP feat:Init commit for rust backend #1180

Conversation

Aisuko commented Oct 17, 2023 • edited

mudler commented Oct 17, 2023

lu-zero commented Oct 31, 2023

Aisuko commented Oct 31, 2023 • edited

Update

lu-zero commented Nov 1, 2023

Aisuko commented Nov 1, 2023

lu-zero Nov 16, 2023

Choose a reason for hiding this comment

Aisuko Nov 17, 2023

Choose a reason for hiding this comment

netlify bot commented Nov 23, 2023 • edited

❌ Deploy Preview for localai failed.

Aisuko Nov 23, 2023

Choose a reason for hiding this comment

lu-zero Dec 21, 2023

Choose a reason for hiding this comment

Aisuko commented Oct 17, 2023 •

edited

Aisuko commented Oct 31, 2023 •

edited

netlify bot commented Nov 23, 2023 •

edited