Auto Labeling Error

[Panda64]

When uploading my TF web model to cloud annotations, I get the following error in console when trying to run auto label on an image:

Unhandled Promise Rejection: Error: The shape of dict['x'] provided in model.execute(dict) must be [1,320,320,3], but was [1,1,1,3].

Any idea how to fix this? I can provide additional info if needed.

My model is a custom trained YOLOv5s (converted to a tfjs web model) with the image size set to 320x320 during training and export.

[bourdakos1]

Hmm it could be that the model is YOLOv5. It should handle any model as long as it just takes an image, but this has only been tested and expected to work with the mobilenet ssd models that were trained with the cloud annotation code samples. The UI uses the js sdk for predictions, so try to see if you can get your model working with that

[Panda64]

When I try using my model with the example react app, I get a similar error of:

Unhandled Rejection (Error): The shape of dict['x'] provided in model.execute(dict) must be [1,320,320,3], but was [1,720,1280,3]

Following clues of this being related to image size, I changed the width and height of the video feed in useWebcam.js from:

width: { ideal: 4096 },
height: { ideal: 2160 },

to

width: { ideal: 320 },
height: { ideal: 320 },

This now results in the slightly different error of:

Unhandled Rejection (Error): The dtype of dict['x'] provided in model.execute(dict) must be float32, but was int32

Looks like I might be one step closer to the root of the issue.

I tried to reconvert the model to match the desired shape of [1,1,1,3], but was only able to get as low as [1,32,32,3] when playing around with the image size parameter.

Still trying to figure this out, but it sounds like the problem is coming from the way that YOLOv5 converts the model to tfjs. Would you suspect the same?

[bourdakos1]

Ah okay, I think the ssd models have a preprocessing step that auto scales the image to match the expected dimensions for the model. The float32 vs int32 error is because the image is passed to the model with pixel values from 0->255 where I'm guessing yolo expects 0->1. Both of these issues should be solvable by adding some adding a preprocessing step to your model

[bourdakos1]

You might also run into issues with the prediction output format, because the sdk expects it to be a certain way and runs non max suppression on the results

[UPstartDeveloper]

Hey @bourdakos1 - I'm another dev from @Panda64's team:

We looked a bit into the kind of preprocessing in Mobilenet, and we're thinking we'll try to do the following to make our model work. We're curious to hear if you'd have any additional thoughts on our approach:

1. Convert the initial model (implemented in PyTorch) to a Tensorflow SavedModel format
2. Add 2 additional layers to the tf model:
   1. A Resizing layer to crop/pad the input images to be what the model expects (e.g. 320x320) (from Keras)
   2. A Lambda layer we can use to standardize the pixel values to fall between [0, 1] (also from Keras)
3. If all goes well, then go about the rest of steps like normal (e.g. converting to tfjs, and uploading it to IBM Cloud Annotations)

[bourdakos1]

That sounds good to me 👍