How to use ffmpeg to upscale video resolution with libplacebo?

[Arc-2023]

Discussed in #199

^{Originally posted by Arc-2023 July 30, 2023}
In Vapoursynth, it is convenient to download and use plugins to change the frame rate of videos. However, it seems that using FFmpeg has more limitations (or maybe I haven't found the right method). Anime4K is a video enhancement algorithm suitable for anime and similar videos. In MPV, their plugins can be imported and used for real-time frame rate conversion.

In FFmpeg, how can I use the algorithms in glsl files to change the video frame rate, instead of using FFmpeg's built-in algorithms?

In FFmpeg, i can use libplacebo filters can also be used to apply GLSL shaders. In my tests, using command like:

libplacebo=custom_shader_path=Anime4K_Upscale_CNN_x2_VL.glsl

this leads to noticeable GPU usage, but the output video resolution remains the same as the original.

I can pass in the w and h parameters when calling the libplacebo filter:

libplacebo=w=iw*2:h=ih*2:custom_shader_path=shaders/Anime4K_Upscale_CNN_x2_VL.glsl

This achieves the purpose of super resolution, but is it essentially using anime4k for upscaling behind the processing?
Is it using the upscaling algorithm in libplacebo first, before applying anime4k? How can I achieve my original goal that only use anime4k to upscale?
here's my full command (in python):

[
    'ffmpeg -hide_banner',
    '-hwaccel', 'cuda',
    '-i', '"{input_ab_path}"',
    '-filter_complex',
    '"[0:v]libplacebo=custom_shader_path=libplacebo=custom_shader_path=shaders/Anime4K_Upscale_CNN_x2_VL.glsl,subtitles="{sub_file}":si=0[out]"',
    '-map', '0:a',
    '-map', '"[out]"',
    '-c:v', 'libsvtav1',
    '-svtav1-params', 'scm=2:scd=1:enable-overlays=1:enable-tf=0:tune=0:preset=7:crf=18',
    '-c:a', 'libvorbis',
    '-qscale:a', '10',
    '-pix_fmt', 'yuv420p10le',
    '-y', '"{output_ab_path}"'
]

[AN3223]

I have an example of using ffmpeg's libplacebo filter here, relevant snippet:

ffmpeg -y -init_hw_device vulkan -i images/anime/mountain_adventurer.webp -vf 'crop=128:128:y=128,format=yuv420p,gblur=0.625,hwupload,libplacebo=w=128:h=128:custom_shader_path=$?,hwdownload,format=yuv420p,scale=256:256:flags=neighbor' $@

($? and $@ are replaced by the shader path and the output filename respectively.)

So you need to specify a vulkan device with -init_hw_device, you need to do hwupload first, you need to do hwdownload after libplacebo, and you may need to specify an appropriate format before and/or after. Also you put libplacebo=custom_shader_path= twice in your command.

Is it using the upscaling algorithm in libplacebo first, before applying anime4k? How can I achieve my original goal that only use anime4k to upscale?

If your output resolution (libplacebo's w and h parameters) is 2x the input resolution and you apply a shader that does a 2x upscale, then the shader should be doing all of the upscaling.

[Arc-2023]

I have an example of using ffmpeg's libplacebo filter here, relevant snippet:

ffmpeg -y -init_hw_device vulkan -i images/anime/mountain_adventurer.webp -vf 'crop=128:128:y=128,format=yuv420p,gblur=0.625,hwupload,libplacebo=w=128:h=128:custom_shader_path=$?,hwdownload,format=yuv420p,scale=256:256:flags=neighbor' $@

($? and $@ are replaced by the shader path and the output filename respectively.)

So you need to specify a vulkan device with -init_hw_device, you need to do hwupload first, you need to do hwdownload after libplacebo, and you may need to specify an appropriate format before and/or after. Also you put libplacebo=custom_shader_path= twice in your command.

Is it using the upscaling algorithm in libplacebo first, before applying anime4k? How can I achieve my original goal that only use anime4k to upscale?

If your output resolution (libplacebo's w and h parameters) is 2x the input resolution and you apply a shader that does a 2x upscale, then the shader should be doing all of the upscaling.

thanks a lot for your answering!

if i don't use -init_hw_device vulkan , 'hwpoload' and 'hwdownload' these three options for transcoding, I noticed some GPU usage. Does this mean shaders are being used for processing?

[AN3223]

Seeing GPU usage is a good sign but you should compare the output to the input to ensure that the shaders are working, you should also see that the output resolution is 2x the input resolution.

[Arc-2023]

Thank you for the reply, but I am still not clear about the specific functions of hwupload and hwdownload. The information I could find only provides simple descriptions of their roles. If I don't add these options, why would the GPU still be occupied? Does this mean adding or not adding these parameters does not affect shader invocation? Or does ffmpeg automatically upload streams to VRAM for processing when handling video and audio streams? If so, what is the significance of manually specifying upload and download?

Also, I noticed that if I directly remove hwdownload, there are still no issues. Does this mean hwdownload is unnecessary?