Segmentation support for models

[Aciid]

Dear ML enthusiasts and Seeed employees.
I trust this message finds you well and thriving.

Describe the feature

Currently, the primary focus of the YOLOv5 -models implementation in SSCMA pertains to object detection, classification, and bounding box predictions tasks.

Yet, there is a substantial potential for the incorporation of segmentation, a concept relating to the identification of pixel-level masks for objects present within an image or video.

In essence, this could facilitate understanding of the extent of the sensor/image occupied by the identified label, its corresponding coordinates and resulting pixel-mask.

Motivation

The implications of this feature are discussed through prototype example use-cases associated with Grove-AI and SenseCap A1101 for Seeed Studios clientele, engrossing interest in both LoRaWAN applications and real-time computer vision, thereby yielding collective advantages. Leaving the implementation and use of the pixel-mask to the firmware gives vast uses to make use of the date on the edge, even in LoRaWAN one can then uplink the solution derived from the pixel-mask. Or uplink the pixel-mask coordinate-path depending on the payload size.

Precision Agriculture: Farming scenarios can leverage segmentation to differentiate lands based on productivity, crop variety, and yield forecasts, capitalizing on sensors capable of extracting pixel-masks, thus presenting pragmatic agricultural process applications.

Industrial Inspection: Experimental laboratory settings may utilize pixel-mask extraction for inspecting PCB wafers before deploying costly machine vision cameras.

Volume Calculation: With a model trained for multiple area and object considerations, pixel-mask detection can be employed provisionally to compute the volumetric occupation of objects.

Smart Cities: Segmentation can validate appropriately sized vehicles occupying specific spaces – for example, confirming an SUV-spot being occupied by a U-Haul truck.

Industry: Large shipping yards and industrial areas can deploy segmentation to enforce safety regulations by detecting objects breaching OSHA rules utilising more advanced computer vision than just object detection.

Remote Sensing: Suitable drones could use pixel-masks to detect and monitor changes in land usage, produce, deforestation, and urban sprawl.

My brief research ( N/A SSCMA )

In preparation for writing to Seeed-Studio, I have examined Seeed-Studio's yolov5-swift fork and made notes on Yolov5’s segmentation support that was added a few months later in the upstream through this sizeable commit - YOLOv5 segmentation model support.
I make no mistake to identify the significant changes and features that Seeed-studios team is maintaining on this fork, with TFLite implementations I understand where we are now and why this has not been a priority

Related resources ( N/A SSCMA )

The upstream feature discussion was very useful to get a briefing on how things were implemented v7.0 - YOLOv5 SOTA Realtime Instance Segmentation #10258
Official announcement Introducing Instance Segmentation in YOLOv5 v7.0

Most importantly the papers on the subject, I found no one specific paper unfortunately. Ultralytics is refering to COCO -implementation on their segmentation model. Instance Segmentation

I made use of Roboflow's documents and notebooks on using their online-tool to segment images dataset for training at What is YOLOv5 Instance Segmentation?. Roboflow is a easy ad-hoc tool, but not required to work on segmentation.

Addenum

After spending more time with SSCMA I think the topic is about mmyolo's segmentation implementations 15_minutes_instance_segmentation.md

I believe these improvements can further bolster the functionalities and applicability of SSCMA and expand its reach across multiple verticals. I am deeply appreciative of your work thus far on yolov5-swift, and I hope to contribute in providing the best proposition through this suggestion.

I eagerly anticipate your feedback regarding the proposal. Following my prior formal submission, I feel compelled to expound upon my perspectives, and share these insights within our community.

Best Regards,

Aciid

[Kway99]

Thanks for reaching out to us! This is Lee from SenseCAP team.
At this stage, the computing power of the AI chip of the A1100 limits the feature. Insufficient computing power cannot support the algorithm of the detecting segment. This is a limitation at the hardware layer, but I will forward this requirement to our product manager and consider it in the future development process.

If you have any further questions or suggestions, please don't hesitate to reach out. We are here to assist you.