Not able to implement yolov8 using the sample pipeline available.

[ParimalMuley]

The sample pipeline I followed.
dabaadc

Hi, I am new to nnstreamer and I am trying to integrate a yolov8s.tflite float16 model in nn which is a pretrained model exported to tflite I tried to implement it using the sample pipeline available but I am facing some issues please help me if anyone could to resolve this issues. This is the pipeline i am working with.

gst-launch-1.0
v4l2src ! videoscale ! videoconvert ! video/x-raw,width=320,height=320,format=RGB,framerate=30/1,pixel-aspect-ratio=1/1 ! tee name=t
t. ! queue ! tensor_converter ! other/tensors,num_tensors=1,types=uint8,format=static,dimensions=3:320:320:1 !
tensor_transform mode=arithmetic option=typecast:float32,add:0.0,div:255.0 !
queue leaky=2 max-size-buffers=2 !
tensor_filter framework=tensorflow2-lite model=./yolov8s_float16.tflite custom=Delegate:XNNPACK,NumThreads:4 latency=1 !
other/tensors,num_tensors=1,types=float32,format=static,dimensions=2100:84:1 !
tensor_transform mode=transpose option=1:0:2:3 !
tensor_decoder mode=bounding_boxes option1=yolov8 option2=./coco-80.txt option3=0 option4=320:320 option5=320:320 !
video/x-raw,width=320,height=320,format=RGBA ! mix.sink_0
t. ! queue ! mix.sink_1 compositor name=mix sink_0::zorder=2 sink_1::zorder=1 ! videoconvert ! autovideosink

I double checked the input output prameters for the tensor filter as well as the decoder to make sure that the datatype is float32.

error message bb_getOutCaps: assertion 'config->info.info[0].type == _NNS_FLOAT32' failed

second pipeline that I tried
Tried another pipeline with the float16 model with input as 320x320 and output dimensions as 1:84:2100 was able to run this pipeline without any warnings or errors but no bounding box output is vivsible. This is the pipeline

gst-launch-1.0
v4l2src ! videoscale ! videoconvert ! video/x-raw,width=320,height=320,format=RGB,framerate=30/1,pixel-aspect-ratio=1/1 ! tee name=t
t. ! queue ! tensor_converter! other/tensors,num_tensors=1,types=uint8,format=static,dimensions=3:320:320:1 !
tensor_transform mode=arithmetic option=typecast:float32,add:-127.5,div:127.5 !
queue leaky=2 max-size-buffers=2 !
tensor_filter framework=tensorflow2-lite model=/home/parimal/Downloads/pari/yolov8s_float16.tflite ! other/tensors,num_tensors=1,types=float32,format=static,dimensions=2100:84:1 !
tensor_transform mode=transpose option=1:0:2:3 ! tensor_transform mode=arithmetic option=typecast:float32,add:-127.5,div:127.5 !
tensor_decoder mode=bounding_boxes option1=yolov8 option2=/home/parimal/yolov8s_saved_model/coco-80.txt option3=0 option4=320:320 option5=320:320 !
video/x-raw,width=320,height=320,format=RGBA ! mix.sink_0
compositor name=mix sink_0::zorder=2 sink_1::zorder=1 ! videoconvert ! autovideosink
t. ! queue leaky=2 max-size-buffers=10 ! mix.

[taos-ci]

cibot: Thank you for posting issue #4456. The person in charge will reply soon.

[myungjoo]

1. Please check the output type/dimensions of tensor_filter framework=tensorflow2-lite model=/home/parimal/Downloads/pari/yolov8s_float16.tflite

   • If this models' output is not float32, and you are applying caps-filter (other/tensors,...) without actually transofming the format, you are creating inconsistent data stream.

2. In the second pipeline, tensor_transform mode=arithmetic option=typecast:float32,add:-127.5,div:127.5 is doing normalization for uint8 (0 - 255) into float (0.0 ~ 1.0). Try tensor_transform mode=typecast,option=float32 instead. With ,add:-127.5,div:127.5, you have distorted it a lot.

3. With Gstreamer debug options, more info will be available for you. (e.g., the input/output dimension/type recognized by tensor_filter, tensor_decoder)

@anyj0527 Any ideas?

[anyj0527]

@ParimalMuley
yolov8 model needs ... tensor_decoder mode=bounding_boxes option1=yolov8 option2=coco-80.txt option3=1:0.65:0.45 .... option3=1 would work for yolov8 models (option3=0 for yolov5 models). Check this.