Gvaclassify signal

[WilliamWright-JCI]

Added signal classify-roi which may be issued for a tracked object (ROI) in one out of every reclassify-interval frames prior to the ROI being classified. The signal includes the metadata for the ROI. A signal subscriber may examine the ROI's metadata, and instruct gvaclassify to skip classfication of the ROI when classification is deemed unnecessary, e.g. ROI dimensions are too small.

The return value from the last signal subscriber will indicate whether or not the ROI should be classified.

To maintain existing behaviour, the classify-roi signal will only be raised when the new property signal-classify-roi is TRUE. By default this property is FALSE.

To allow classification to be unaffected when there is no subscriber, a signal handler return value of FALSE indicates that classification should be run, and a return value of TRUE indicates that classification should be skipped.

As signal generation and handling is likely to be more expensive than examining variables local to gvaclassify, the emission of the signal is:

• delayed until it has been confirmed that classification will not be skipped due to the value of the reclassify-interval property.
• made before various classification history variables are set when classification would ordinarily have occurred (see ClassificationHistory::IsROIClassificationNeeded)

[nnshah1]

Thanks for the pull request - indeed an interesting and useful feature. We've done similar things using pad_probes / gyapython - to intercept frames as they enter gvaclassify and have access to the metadata. The rest of the team will need to weigh in but generally speaking - do you see this as generally useful outside of the example given (i.e. roi size)? ROI size specifically could be added as an additional property to filter regions (as we filter based on detection label). Obviously that is not as flexible as adding the signal - but if that is the primary use - might be worth a dedicated property.

To allow classification to be unaffected when there is no subscriber, a signal handler return value of FALSE indicates that classification should be run, and a return value of TRUE indicates that classification should be skipped.

Is there a way to do the opposite? To keep consistent with the behavior of gvapython would like FALSE to indicate a dropped region, and TRUE to indicate to go ahead and process. I agree if no subscriber is there then it should always be TRUE.

[mattg-sp]

Hello. I work with William and actually suggested this feature. So, perhaps it's best if I explain the underlying rationale.

We're porting some existing code to utilize the gvaclassify element, for performing object-based inferencing. Our prior implementation conditioned it on the basis of:

• Object class - one of potentially several classes supported by our particular model.
• Group status - our tracker could generate the equivalent of RoIs that contained multiple objects. We wanted to avoid classifying such groups.
• Meets or exceeds minimum size threshold.
• Quasi-exponential frequency backoff - unlike gvaclassify, ours uses timestamps to avoid presuming a fixed framerate, and runs at a variable frequency that approaches zero, the longer the object remains visible.

However, I can easily think of several other criteria on which to predecate classification:

• Detection confidence - perhaps only reclassifying when the object is detected with a higher confidence than the last time classification was run. This could make sense in cases where classification accuracy correlates with detection confidence.
• Based on the confidence of the last time it was run on this object - perhaps only reclassifying if the previous result had low confidence.
• Based on the result of another classification model. One such example might be to use an upstream classifier to deterimne the object's orientation. In this case, maybe you only want to inference when seeing a new side of the object.
• Maximum size - objects too close to the camera could get distorted to the point where accuracy is compromised.
• The region's aspect ratio - one might wish to exclude objects that are significantly obstructed.

So, one can imagine a wealth of scenarios. What's more, because multiple handlers can be registered for the same signal - any one of which can short-circuit evaluation of the others - they provide a natural mechanism by which one can compose more sophisticated criteria, perhaps even by drawing some from a library of pre-written handlers.

Finally, for your built-in conditions, you could simplify your code by harnessing the signal's "class handler", which is an automatically-registered handler that's stored in the baseclass.

[WilliamWright-JCI]

nnshah1, in response to your question about the following statement:

To allow classification to be unaffected when there is no subscriber, a signal handler return value of FALSE indicates that classification should be run, and a return value of TRUE indicates that classification should be skipped.

My knowledge of GLib is limited, but I believe that there is no way to do the opposite if a gboolean is used as the return value. The documentation for g_signal_emit() states the following:

Note that g_signal_emit() resets the return value to the default if no handlers are connected...

So, if there are no connected handlers the gboolean return value from signal_roi return value will be FALSE. And in this situation, we want classification to go ahead.

[mattg-sp]

With all that being said, I'm concerned that this submission might be a bit premature. First, as I wrote that comment, I consulted the docs and noticed that the patch registers the signal with a NULL accumulator. According to the docs, that means "the return value of signal emissions is then the value returned by the last callback", which I think is not what we want. Furthermore, I checked the test and do not see evidence that we tried using multiple handlers.

See: https://developer.gnome.org/gobject/stable/gobject-Signals.html#GSignalAccumulator

Instead, it seems we might do well to look into using g_signal_accumulator_true_handled().

Second, a concern has been raised about the runtime overhead of simply emitting the signal. The property which enables/disable signal emission shouldn't be needed, since signals are little more than just callbacks that can have multiple subscribers. When I sought to quantify their overhead, I found it took about 34 ns, on a i7-6700. That was just using a trivial pipeline, however (utilizing just fakesrc, fakesink, and identity).

Separately, William tested according to a different methodology and reported that he found the overall CPU utilization of a pipeline involving gvadetect, gvatrack, gvaclassify, and gvawatermark went up by a whopping 0.4% (all-core), just from merely enabling the signal. Until we either invalidate that measurement or confirm it and understand/address the root cause, I think it would be worth holding off on taking this patch.

Thanks for your interest and your work on dlstreamer.

[mattg-sp]

@WilliamWright-JCI I would suggest the name of the signal is misleading, for a return value of FALSE. IMO, it should probably be called something like skip-roi.

However, we should confirm proper handling of multiple subscribers, before finalizing that.

[mattg-sp]

This basically just reiterates my previous two comments.

[mattg-sp]

Consider renaming to something like "should-skip-roi", since that's the effect of a handler returning TRUE.

[mattg-sp]

The accumulator parameter should probably be &g_signal_accumulator_true_handled. We need to actually test that with multiple handlers, as well.

[mattg-sp]

If we can address or otherwise eliminate the apparent overhead, then this property can be eliminated.

The 34 ns I measured is at least 5 orders of magnitude below the typical cost of classification. So, we should be fine to always emit the signal, if it turns out that the "0.4% increase in CPU time" was either a measurement error or a real performance problem we can fix.

[nnshah1]

@mattg-sp Thanks for expanding on different use cases - they all seem valid and interesting.

Maybe you can help in defining a list of "filter requirements". I think after seeing that we can also look at different potential ways to allow for that flexibility. The signal / callback approach is one way, another may be a "filter" element.

Have you tried implementing any similar filtering capabilities via the DL Streamer VideoFrame api (via pad probes or gvapython)? I wonder what the performance hit would be for putting it in line as an element vs a signal.

(That may be a silly question as my experience with the performance comparison of signals to elements is close to zero).

Are you already working with someone on the DL Streamer / OpenVINO team? It might be helpful for us to learn more about your use case and requirements and we could set up a time to meet (assuming that makes sense to you). There are also some upcoming features that may have an impact on the filtering capabilities as well.

Thanks for your interest and work on DL Streamer :)

Neelay

[intel-neena]

@mattg-sp and @WilliamWright-JCI We are working on the feature enhancement related to #57 which might be impacted by this pull request. We would like to understand your PR better. If you send me an email, I can reach out to you and set up a meeting.

[pbochenk]

Hi @WilliamWright-JCI and @mattg-sp, thanks for PR and sorry for delay reviewing it! We had long debates discussing two options - signals or metadata field for this purpose, and decided to go with metadata option. We will include this capability into future releases.