Wlfreerdp uanble to use hardware acceleration

[S-Bhavin]

I am using wlfreerdp to access remote desktop and render it on weston on rockchip (rk3328 SoC). Now, what I have been trying to do is get wlfreerdp to use hardware accelerator so that it doesn't loads up the cpu.

For this reason I have built FreeRDP (version-2.6.1) using yocto (kirkstone) with -DWITH_H264=ON , -DWITH_WAYLAND=ON, -DWITH_GSTREAMER_1_0=ON, -DWITH_VULKAN=ON, -DWITH_WINPR=ON, -DWITH_SWSCALE=ON, -DWITH_FFMPEG=ON, -DWITH_VAAPI=OFF configurations.

Also I have built FFMPEG (version-6.1) with --enable-libdrm, --enable-shared, --enable-v4l2-request. --enable-libudev, --enable-sdl2
configurations.

I tested FFMPEG separately with -hwaccel and got the logs of VPU (HANTRO) being used during that test. Below are the commands that I used to test ffmpeg and their results :

root@rockpi-e-rk3328:~# ffmpeg -hwaccels
ffmpeg version n6.1.1-21-g9714209fb0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11.4.0 (GCC)
  configuration: --cross-prefix=aarch64-poky-linux- --ld='aarch64-poky-linux-gcc -mcpu=cortex-a53 -march=armv8-a+crc+crypto -mbranch-protection=standard -b
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
Hardware acceleration methods:
drm

root@rockpi-e-rk3328:~# ffmpeg -hwaccel drm -i BBB-720p-30-fps.mp4 -f null - -benchmark
ffmpeg version n6.1.1-21-g9714209fb0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 11.4.0 (GCC)
  configuration: --cross-prefix=aarch64-poky-linux- --ld='aarch64-poky-linux-gcc -mcpu=cortex-a53 -march=armv8-a+crc+crypto -mbranch-protection=standard -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/recipe-sysroot' --cc='aarch64-poky-linux-gcc -mcpu=cortex-a53 -march=armv8-a+crc+crypto -mbranch-protection=standard -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/recipe-sysroot' --cxx='aarch64-poky-linux-g++ -mcpu=cortex-a53 -march=armv8-a+crc+crypto -mbranch-protection=standard -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/recipe-sysroot' --arch=aarch64 --target-os=linux --enable-cross-compile --extra-cflags=' -O2 -pipe -g -feliminate-unused-debug-types -fmacro-prefix-map==/usr/src/debug/ffmpeg/5.0.1-r0 -fdebug-prefix-map==/usr/src/debug/ffmpeg/5.0.1-r0 -fdebug-prefix-map=/recipe-sysroot= -fdebug-prefix-map=/recipe-sysroot-native= -mcpu=cortex-a53 -march=armv8-a+crc+crypto -mbranch-protection=standard -fstack-protector-strong -O2 -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security --sysroot=/recipe-sysroot' --extra-ldflags='-Wl,-O1 -Wl,--hash-style=gnu -Wl,--as-needed -fmacro-prefix-map==/usr/src/debug/ffmpeg/5.0.1-r0 -fdebug-prefix-map==/usr/src/debug/ffmpeg/5.0.1-r0 -fdebug-prefix-map=/recipe-sysroot= -fdebug-prefix-map=/recipe-sysroot-native= -Wl,-z,relro,-z,now' --extra-libs=-lm --sysroot=/recipe-sysroot --libdir=/usr/lib --shlibdir=/usr/lib --datadir=/usr/share/ffmpeg --cpu=cortex-a53 --pkg-config=pkg-config --enable-libdrm --enable-shared --enable-v4l2-request --enable-libudev --enable-hwaccel=libdrm --disable-static --enable-alsa --disable-altivec --enable-avcodec --enable-avdevice --enable-avfilter --enable-avformat --enable-bzlib --disable-libfdk-aac --disable-gpl --disable-libgsm --disable-indev=jack --enable-libdrm --disable-libopus --disable-libvorbis --enable-lzma --disable-libmfx --disable-mipsdsp --disable-mipsdspr2 --disable-libmp3lame --disable-openssl --enable-pic --enable-postproc --enable-pthreads --enable-sdl2 --enable-shared --disable-libspeex --disable-libsrt --disable-stripping --enable-swresample --enable-swscale --enable-libtheora --disable-vaapi --disable-vdpau --disable-libvpx --disable-libx264 --disable-libx265 --enable-libxcb --enable-outdev=xv --enable-zlib
  libavutil      58. 29.100 / 58. 29.100
  libavcodec     60. 31.102 / 60. 31.102
  libavformat    60. 16.100 / 60. 16.100
  libavdevice    60.  3.100 / 60.  3.100
  libavfilter     9. 12.100 /  9. 12.100
  libswscale      7.  5.100 /  7.  5.100
  libswresample   4. 12.100 /  4. 12.100
avcodec_alloc_context3,155
avcodec_alloc_context3,155
id->27---BDEBUG
id->86018---BDEBUG
id->27---BDEBUG
id->86018---BDEBUG
avcodec_open2,134
id->86018---BDEBUG
avcodec_open2,134
avcodec_alloc_context3,155
id->86018---BDEBUG
avcodec_alloc_context3,155
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'BBB-720p-30-fps.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    creation_time   : 2017-03-29T17:07:33.000000Z
  Duration: 00:10:34.60, start: 0.000000, bitrate: 2236 kb/s
avcodec_alloc_context3,155
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 2041 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
avcodec_alloc_context3,155
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 191 kb/s (default)
    Metadata:
      creation_time   : 2017-03-29T17:07:38.000000Z
      handler_name    : IsoMedia File Produced by Google, 5-11-2011
      vendor_id       : [0][0][0][0]
avcodec_alloc_context3,155
id->135169---BDEBUG
avcodec_alloc_context3,155
avcodec_open2,134
avcodec_alloc_context3,155
id->65536---BDEBUG
avcodec_alloc_context3,155
avcodec_open2,134
id->0---BDEBUG
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
  Stream #0:1 -> #0:1 (aac (native) -> pcm_s16le (native))
Press [q] to stop, [?] for help
**[h264 @ 0xaaaac9e2a9a0] Using V4L2 media device /dev/media0 (hantro-vpu) for S264**
avcodec_open2,134
avcodec_open2,134
Output #0, null, to 'pipe:':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isommp42
    encoder         : Lavf60.16.100
avcodec_alloc_context3,155
  Stream #0:0(und): Video: wrapped_avframe, nv12(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 wrapped_avframe
avcodec_alloc_context3,155
  Stream #0:1(und): Audio: pcm_s16le, 44100 Hz, stereo, s16, 1411 kb/s (default)
    Metadata:
      creation_time   : 2017-03-29T17:07:38.000000Z
      handler_name    : IsoMedia File Produced by Google, 5-11-2011
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 pcm_s16le
[out#0/null @ 0xaaaac9b66dc0] video:583kB audio:7144kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
frame= 1243 fps= 43 q=-0.0 Lsize=N/A time=00:00:41.44 bitrate=N/A speed=1.43x    
bench: utime=27.839s stime=1.790s rtime=28.983s
bench: maxrss=97304kB
Exiting normally, received signal 2.

Witnessing the logs of VPU I was able to deduce that FFMPEG is able to use hardware acceleration, but when I access remote desktop using wlfreerdp it doesn't . I added some debug prints and found that wlfreerdp is using ffmpeg but not hardware accelerator.
Below are the logs of that:

root@rockpi-e-rk3328:~# wlfreerdp /u:xxx /p:xxx /v:xxx.xxx.xxx.xxx /gfx-h264:AVC444  /f
wlfreerdp_run,464 --BDEBUG
freerdp_connect,162 --BDEBUG
freerdp_connect,204 --BDEBUG
rdp_client_connect,241 --BDEBUG
rdp_client_reset_codecs,194 --BDEBUG
freerdp_client_codecs_prepare,32 --BDEBUG
freerdp_client_codecs_prepare,44 --BDEBUG
freerdp_client_codecs_prepare,56 --BDEBUG
freerdp_client_codecs_prepare,68 --BDEBUG
freerdp_client_codecs_prepare,80 --BDEBUG
freerdp_client_codecs_prepare,96 --BDEBUG
freerdp_client_codecs_prepare,111 --BDEBUG
h264_context_new,578 --BDEBUG
h264_context_init,533 --BDEBUG
h264_context_init,542 --BDEBUG
h264_register_subsystems,497 --BDEBUG
h264_register_subsystems,521 --BDEBUG
libavcodec_init,501 --BDEBUG
id->27---BDEBUG
avcodec_alloc_context3,155
avcodec_open2,134
[06:38:33:407] [459:459] [WARN][com.freerdp.crypto] - Certificate verification failure 'self-signed certificate (18)' at stack position 0
[06:38:33:407] [459:459] [WARN][com.freerdp.crypto] - CN = DESKTOP-IH221QF
[06:38:34:723] [459:459] [INFO][com.freerdp.gdi] - Local framebuffer format  PIXEL_FORMAT_BGRA32
[06:38:34:724] [459:459] [INFO][com.freerdp.gdi] - Remote framebuffer format PIXEL_FORMAT_BGRA32
sh: xprop: command not found
sh: xprop: command not found
[06:38:34:847] [459:459] [INFO][com.freerdp.channels.rdpsnd.client] - [static] Loaded fake backend for rdpsnd
[06:38:34:848] [459:459] [INFO][com.freerdp.channels.drdynvc.client] - Loading Dynamic Virtual Channel rdpgfx
h264_context_new,578 --BDEBUG
h264_context_init,533 --BDEBUG
h264_context_init,542 --BDEBUG
libavcodec_init,501 --BDEBUG
id->27---BDEBUG
avcodec_alloc_context3,155
avcodec_open2,134
h264_context_new,578 --BDEBUG
h264_context_init,533 --BDEBUG
h264_context_init,542 --BDEBUG
libavcodec_init,501 --BDEBUG
id->27---BDEBUG
avcodec_alloc_context3,155
avcodec_open2,134

The above command does access the remote desktop server but it lags over the display. The CPU usage exceeds over 100% and the rendered desktop keeps blinking. My guess is, it is because of hardware acceleration not being used.

So I want to know if wlfreerdp does support hardware acceleration. If it does, then what could I be doing wrong here?

[akallabeth]

check libfreerdp/codec/h264_ffmpeg.c
currently there is a path for VAAPI support in there, but there might be a flag missing for other accelerators.

[S-Bhavin]

Thanks for the reply @akallabeth ,

check libfreerdp/codec/h264_ffmpeg.c currently there is a path for VAAPI support in there, but there might be a flag missing for other accelerators.

Which flag for other accelerators?? because the only flag that I can see in libfreerdp/codec/h264_ffmpeg.c is WITH_VAAPI
With reference to h264_ffmpeg.c, does it mean the implementation of freerdp requires vaapi for hardware acceleration or there is another way around for other hw accelerators?

I am using lima driver provided by mesa for the support of MALI450 gpu.
If I have to work with vaapi will there be any role of lima and mesa? If yes, what is the dependency?

Basically I have following pieces working with me:

1. FFMPEG -> able to use hardware acceleration using HANTRO VPU when tested individually
2. LIMA + MESA -> able to provide smooth graphical rendering using MALI450 GPU over weston (wayland).
3. Gstreamer -> able to use HANTRO VPU for video rendering. (gstreamer pipeline :- gst-launch-1.0 filesrc location=/home/root/BBB-720p-30-fps.mp4 ! qtdemux ! v4l2slh264dec ! waylandsink fullscreen=true)
4. FREERDP -> has support of ffmpeg but isn't using hardware acceleration. (both wlfreerdp & xfreerdp).

Am I missing something in configuring freerdp or in understanding?

I want to integrate these individual pieces to get FREERDP to somehow work with hardware acceleration. Any possibilities to accomplish this using aforementioned components?? OR some other way around??

[akallabeth]

@S-Bhavin there is currently only the VAAPI version implemented in that file, but others can be added.
you might check the source of the FFMPEG tools on how to do that.

[S-Bhavin]

Thanks for the reply @akallabeth,
Would like if you could help with the other questions mentioned above as well.

[akallabeth]

• there is no gstreamer implementation in FreeRDP (only the deprecated TSMF channel did use that)
• rendering is up to the client (xfreerdp -> X11, sdl-freerdp -> SDL (X11 or Wayland))

[S-Bhavin]

Hi @akallabeth,
I have got GPU working with the opensource LIMA driver provided by mesa. Now the rendering is smooth and the cpu load isn't much as well. I tested it with glmark2-es2-wayland and the overall performance is good.

But when I am playing any mp4 video, it plays using cpu and loads it up. Now if I use some older version supporting TSMF, that can use gstreamer, do you think it can help taking off the load from cpu? If not, what is advisable to play video without loading up the cpu too much?

[akallabeth]

@S-Bhavin well, just as I told you, patch the file above for support of these accelerators?
and no, TSMF is useless.

[S-Bhavin]

Hi @akallabeth
I am willing to make changes in freerdp to support ffmpeg, but I am new at freerdp and ffmpeg so can you guide me with the changes (like macros or functions to be called to integrate ffmpeg with freerdp) that are supposed to be made in order to achieve this.

Could really use some help for this modification.

[akallabeth]

@S-Bhavin

1. WITH_VAAPI is already integrated with an option (although unstable)
2. ./libfreerdp/codec/h264_ffmpeg.c already implement ffmpeg support, but that might need to be adjusted to utilize hardware accelerators
3. you might want to join our matrix channel (see https://freerdp.com for details) if you have further questions.