Looking for advice on how to integrate into gstreamer glfilterapp client-draw callback

[iongion]

I am looking for a way to integrate thorvg as a drawing canvas for gstreamer - to allow programmatic graphics to be added into a video processing pipeline.

gstreamer, through a filter like glfilterapp that can expose a custom draw-callback, where instead of dealing with gl primitives, I would prefer to use the high level api of thorvg.

An example of a draw callback is available here https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/-/blob/21dd771b69b5c90ee213521f83a4fb4bcb23b0a1/tests/examples/gl/generic/recordgraphic/main.cpp#L68

I am looking for some advice on how I can integrate thorvg with an existing opengl context / texture

[hermet]

@RuiwenTang hello, do you have any opinion?

[iongion]

I have attached where I am with my exploration, basically it is a butchered gltestsrc filter

• https://github.com/GStreamer/gst-plugins-base/blob/master/ext/gl/gstgltestsrc.c
• https://github.com/GStreamer/gst-plugins-base/blob/master/ext/gl/gltestsrc.c

I am trying to integrate with thorvg as a new testsrc filter pattern

gst-production-overlay.zip

Starting up

1. Fetches thorvg and builds locally - ./build-thorvg.h
2. Build gstreamer plugin library - ./build-overlay.sh
3. Run the pipeline using ./develop.sh to see the filter in action

The thorvg functions are in gst-production-overlay/gst-production-overlay-libs/ext/gl/gloverlaydraw.cpp
My knowledge of opengl is rock bottom, it is just experimentation.

Learned that gstreamer opengl support is abstracted, to force gles/egl, these variables are set in the build & develop scripts

export GST_GL_PLATFORM=egl
export GST_GL_API=gles2 

I am able to see the red ellipsis being drawn, but it does not keep between redraws, it is just at first draw and then the filter is black pixels (due to that clear call maybe)

[hermet]

Thank you for the information. Currently, ThorVG's GL support is in an experimental state, we also need to test its compatibility and functionality.

You might want to try using the latest (main) branch, where relevant modifications have been made. For instance, you can use:

canvas->target(0, width, height)

The updates would draw the things on to the main surface (not fbo). atm, I'm not certain it works or not, but at least I think you can test it easily.

Also, I have a question:

• Could you explain what the 'texture' parameter contains in the filter function? Is it possible to upload an image to that texture?

p.s I look forward to revisiting this soon for a review in detail, once I find an appropriate time.

[iongion]

Thanks for the tips, I will give them a shot.

As for the texture parameter, i think so, gstreamer has an example here where they load an image to texture

https://github.com/GStreamer/gst-plugins-base/blob/master/ext/gl/gstgloverlay.c

[hermet]

"If the texture is ready for users, you might be able to draw shapes with SwCanvas and upload the image there. Although it's not optimal, I believe it should work perfectly.

[iongion]

Awesome tip, will continue the experiment at least, in the end I want this to run on mobile devices and apply overlay to their 4k recording cameras.

[iongion]

Yaayyy, the circle is there, awesome @hermet - using now the main branch of the repo and adapted the initializer

[stevenlstarr]

It's not entirely clear what your project requirements are, but if you're not specifically tied to using GStreamer, you might consider using libVLC (https://www.videolan.org/vlc/libvlc.html) instead. libVLC is generally easier to use for handling video playback and processing tasks. Whether you choose libVLC or GStreamer, the key is to extract individual frames into a buffer. Once you've got the frames in a buffer, you can perform various compositing operations using a graphics library of your choice, such as ThorVG, Skia, GraphicsMagick, GLSL shaders, or OpenCV.

For example, you could mask the video frames to shape the video output like a teddy bear, and even add particle effects shooting out from the teddy bear's ears. And add a clown wig as an overlay.

Picture.load supports raw data from a memory block with a given size.
pseudocode

#include <vlc/vlc.h>
#include <ThorVG.h>
#include <iostream>

// Frame capture callback (simplified; consider frame format and actual implementation)
void* lock(void* data, void** p_pixels) {
    // Assuming frame format is directly usable or converted to ARGB
    auto frameBuffer = new unsigned char[width * height * 4]; // Example, width and height must be defined
    *p_pixels = frameBuffer;
    return nullptr;  // Or your actual frame context
}

void unlock(void* data, void* id, void* const* p_pixels) {
    unsigned char* pixels = reinterpret_cast<unsigned char*>(*p_pixels);
    // Convert and process the frame if necessary
}

int main() {
    // Initialize libVLC
    libvlc_instance_t* vlcInstance = libvlc_new(0, nullptr);
    libvlc_media_player_t* mediaPlayer = libvlc_media_player_new(vlcInstance);
    libvlc_media_t* media = libvlc_media_new_path(vlcInstance, "video.mp4");
    libvlc_media_player_set_media(mediaPlayer, media);
    libvlc_video_set_callbacks(mediaPlayer, lock, unlock, nullptr, nullptr);
    libvlc_media_player_play(mediaPlayer);

    // Initialize ThorVG
    tvg::Initializer::init(tvg::CanvasEngine::Sw, 0);

    // Create a canvas
    auto canvas = tvg::SwCanvas::gen();

    // Create a picture and load frame data into it
    auto picture = tvg::Picture::gen();
    picture->load(reinterpret_cast<uint32_t*>(*p_pixels), width, height, false);

    // Create a mask and apply it
    auto mask = tvg::Shape::gen();
    mask->appendCircle(250, 325, 225, 225);
    mask->fill(255, 255, 255, 255);
    picture->composite(move(mask), tvg::CompositeMethod::AlphaMask);

    // Push the picture into the canvas
    canvas->push(move(picture));

    // Draw the canvas
    canvas->draw();

    // Cleanup
    libvlc_media_player_stop(mediaPlayer);
    libvlc_media_player_release(mediaPlayer);
    libvlc_media_release(media);
    libvlc_release(vlcInstance);
    tvg::Initializer::term(tvg::CanvasEngine::Sw);

    return 0;
}