improving the selection of chroma scaler in libplacebo.

[mightyhuhn]

this is pretty much the same request as the one on mpv that did go no where.

so here i try again with hopefully better arguments:
chroma scaling in libplacebo is not up to par to other renderers.

using an external shader to upscale the entire image works just fine. with chroma stuff it get's more complicated with placement and such. so that's very user unfriendly.

so the usual answer will be to use a form of biliteral which can go utterly wrong. (i can proof that yet again but i really don't want to)
so here is an normal example: https://slow.pics/c/nYnRAWTf
shows super xbr 150 which is similar in speed to jinc in madVR. (sorry it is so fast i can not measure it.)
here is another test with upscaling to 4K all scaler are madVR the image scaler was also super xbr 100 in this case:
https://slow.pics/c/zUsy6eUa
this is literally a day night difference: but i had to give up on this after reading this: mpv-player/mpv#12384
reconstruct is very similar to krig biliteral but if that fails it fails even harder then that. so the results is not that interesting to me.

i used this input string to toggle in mpv: CTRL+9 cycle-values cscale "bilinear" "sinc" "jinc" "bicubic" "ewa_lanczos4sharpest" "ewa_robidouxsharp"
why they all look pretty much the same except for bilinear is beyond me and not why i'm here.
there is no GLSL port of super xbr i know of so no mpv port with hooks to. so the best i can poin to is this:
https://github.com/zachsaw/MPDN_Extensions/blob/master/Extensions/RenderScripts/Super-xBR/super-xbr.hlsl

video source: at the end

if there are other ways to get comparable results in gpu-next libplacebo i'm all ears.
have a nice day.

a yeah i forgot pretty sure JRVR has it so there is maybe even a port out there already it's MIT so maybe not.

[haasn]

mpv-player/mpv#12163 (comment)

Edit: I see you're active in that issue as well. I don't think anything has changed? Why don't you just use the (deleted) superxbr-chroma implementation from that repo?

[mightyhuhn]

i'm not only active i was the creator of that now closed issue.

as far as i can see it is gone on that repo. and it needs an x scale to correct positioning without it it's pretty useless and that's why i hate the idea of user based chroma scaler there is to much that can go wrong.
i can't create and write that.
was that ever been able to do chroma?

super xbr was made for pixel art where it does terrible in my honest opinion. maybe the version in madVR is different or the AR is just tuned to perfection and this is not a port of mpdn version. making this a huge waste of time.

super xbr pretty much outperforms jinc in quality while been cheaper.

but i'm not alone in that super xbr is very useful boat.

[haasn]

And did you try RAVU chroma as recommended?

[mightyhuhn]

if it has been updated with //!OFFSET ALIG i could. i'm actually open to test everything but the main point is the missing or lack of shader that are actually can be used with offset correction.

before we run in circles this is where it pretty much ended: mpv-player/mpv#12163 (comment)

i don't even know what that caveats is he is talking about.

[kasper93]

This is more of a discussion https://github.com/haasn/libplacebo/discussions than an issue. I think there are no built-in scalers in libplacebo that would satisfy you and in general I don't see libplacebo integrating meme shaders natively.

I think there are few options of chroma scaling that may be interesting for you:
https://github.com/Artoriuz/glsl-chroma-from-luma-prediction
https://github.com/Artoriuz/glsl-joint-bilateral
https://github.com/bjin/mpv-prescalers (if you dig into commit history you can find older super-xbr, RAVU for chroma...)

[mightyhuhn]

same issue as before chroma placement.
btw. here are my own creation for testing chroma placement: https://drive.google.com/file/d/19wIkcOpBGCtaCeh-ZNsShlCn-FvKnc48/view?usp=drive_link and mpv support 3 which is pretty much one more then you realistically needs and two more then madvr...

why is a cheaper better scaler a meme?
i mean it it doesn't cost more then jinc. my 4060 is doing 1080p to 4K at 30% at 205-400 mhz using 7 watt.

but sure here we go even more comparisons: https://slow.pics/c/tOaGmWhi
nearly for sure not 100 % the same source image i do not know how to frame perfect seek in mpv yet. so all mpv and all madVR are the same source but that it. both only used "jinc" i guess with AR for image scaling. there are even better test image and a better image scaler may show bigger differences too.

ngu AA sharp are a meme as chroma scaler super xbr is not. and my issue with luma guided still stays.
super XBR destroys ewa_sharp and it could be faster.

and yes this is my first try and i sadly manage to break joint already.
i can later go and learn how to go through the history stuff and add ravu chroma and that version of super xbr that supposedly terrible.

[kasper93]

i can later go and learn how to go through the history stuff and add ravu chroma and that version of super xbr that supposedly terrible.

ravu-chroma is here https://github.com/bjin/mpv-prescalers/tree/cc02ed95c1fe05b72bc21d41257c4c085e6e409b
superxbr-chroma is here https://github.com/bjin/mpv-prescalers/tree/1fb1c079405b674d54206ff2c52fd520db63ff10

EDIT: here are all scripts to generate https://github.com/bjin/mpv-prescalers/tree/source

If you want community to look more into those or other solutions, you would need to provide examples where it is better. Currently bilateral variants are commonly agreed to be better.

[mightyhuhn]

at least i can now see why it didn't take off...
https://slow.pics/c/QJOn0psT
this version is pretty much pointless on this test at least. most likely in general.
because they all hands down loose to mv super xbr i'm maybe just doing all of this wrong.
they are loaded using input.conf this way to try stuff:

# Optimized shaders for higher-end GPU
CTRL+1 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Restore_CNN_VL.glsl;~~/shaders/Anime4K_Upscale_CNN_x2_VL.glsl;~~/shaders/Anime4K_AutoDownscalePre_x2.glsl;~~/shaders/Anime4K_AutoDownscalePre_x4.glsl;~~/shaders/Anime4K_Upscale_CNN_x2_M.glsl"; show-text "Anime4K: Mode A (HQ)"
CTRL+2 no-osd change-list glsl-shaders set "~~/shaders/ravu-zoom-r2-chroma.hook"; show-text "ravu r2"
CTRL+3 no-osd change-list glsl-shaders set "~~/shaders/ravu-zoom-r3-chroma.hook"; show-text "ravu r3"
CTRL+4 no-osd change-list glsl-shaders set "~~/shaders/superxbr-chroma.hook"; show-text "super xbr"
CTRL+5 no-osd change-list glsl-shaders set "~~/shaders/CFL_predition.glsl"; show-text "predition"
CTRL+6 no-osd change-list glsl-shaders set "~~/shaders/jointbiliteral.glsl"; show-text "joint"
CTRL+7 no-osd change-list glsl-shaders set "~~/shaders/FSRCNNX_x2_8-0-4-1.glsl"; show-text "8"
CTRL+8 no-osd change-list glsl-shaders set "~~/shaders/FSRCNNX_x2_16-0-4-1.glsl"; show-text "16"
CTRL+9 cycle-values cscale "bilinear" "sinc" "jinc" "bicubic" "ewa_lanczos4sharpest" "ewa_robidouxsharp"
CTRL+0 no-osd change-list glsl-shaders clr ""; show-text "GLSL shaders cleared"

The ported shaders (in mpv) also include contributions licensed under terms of
LGPLv2 or later (particularly, major part of superxbr was rewritten by
@haasn).
interesting is this maybe super xbr 25 and i have to add the difference to the two -1.25 variables to get a different version of it? also i looked into chroma version and search for anti ringing and nothing.

screenshot-high-bit-depth=no is now been used sorry about that but 16 bit screenshots is very cool but as a default maybe a bit over kill.

[Artoriuz]

Just a reminder that most (all?) chroma shaders upsample chroma to luma resolution and not to the final output resolution, so, assuming you're still using this as your test image, you're doubling the merged planes after doubling chroma which kinda invalidates the entire comparison. I tested this exact same frame here and to be honest the shaders seem fine enough: https://slow.pics/c/RlkFmHRb

I do understand your disdain with luma-guided approaches as they can produce subpar results when there's no correlation between the planes, and in that case the best option available is probably RAVU chroma. CfL has some logic to prevent it from producing pure garbage but it's obviously not perfect and it can be easily fooled by things like chromatic aberration.

To me this sounds more like a feature request, but as kasper mentioned you'd have to convince the community this is actually better than what we already have, and then find someone actually willing to work on it.

[mightyhuhn]

first of all your test is amazing it is showing my issue i'm complaining about. the CTL has chroma bleeding most likely because of wrong placement which could explain the heavy aliasing in CTL and something that is fixable.

the doubling of the merged plane is intentional and is fair as long as the doubling is the same scaler. because i want a realistic output. there are multiply reasons for that one is watching 1080p images on UHD is not that fun. the source for this was SD: https://slow.pics/c/tOaGmWhi

does mpv not work like this and will not use "scale" to get to the final destination and will use something else? if that's the case it would start to explain a lot. but in the OSD it looked like it does it this way.
but 1080 has been done by me too where only Chroma is compared: https://slow.pics/c/nYnRAWTf
the results still speak for them self and i don't see how this is not convincing. i know how to make samples: i have this one and it pretty bad: https://drive.google.com/file/d/12YUZbkUqkCGHcU25Ri_bCV0P5PaliDDU/view?usp=drive_link
i have a newer one to that is also not that great: https://drive.google.com/file/d/1E7-XBHJ6Jlndafm5URsxCU3z-oAmZolW/view?usp=drive_link
but better

i'm not confident in my use of mpv at all and the fact that all results are barely different to bilinear proofs that. the super xbr i load there is different but just nothing. does this hook loading even work the image changes slightly so i guess so?

all these comparisons are very time consuming and i'm still not sure they are take serious.

[Artoriuz]

The "normal" order of operations is to upsample chroma to luma resolution, merge the planes, convert them to RGB and then do the final scaling step to the target resolution. You can control which filters to use with scale, cscale and dscale, but generally speaking the defaults should be good enough unless you know what you're doing.

This image may or may not be up to date, but it can give you an idea:

About the filters looking the same, I don't know what you're doing wrongly but they certainly don't: https://slow.pics/c/iH9cifkK
Also, your "superxbr" sample is very blurry which indicates there's more than just chroma scaling happening, the gamma curve also seems to be different as there's clearly some contrast boost in the image.

You're correct though that both CfL and Krig seem to have some negligible pixel shifting in this example, this probably has something to do with the //!OFFSET ALIGN directive but I'm not sure if it's a legitimate issue or just the shaders being too aggressive.

[mightyhuhn]

thanks. with CTL it may just correct the luma channel instead of the chroma channel. but i still see red getting into pure black which it typical for chroma bleeding.
i will make a proper sample to make sure it is the same frame. but good to see that it is pretty much the same as madVR.

i used the AV1 version or maybe that is a key/I frame and the one i used in the madVR is not having better quality.

my madVR is not doing any calibration i have an calibrated end device so colors should be the same...

[mightyhuhn]

scaler in deep: showing what they are actually doing even through you will nearly never ever see it like this:
https://slow.pics/c/ejSmXG85

a bunch of images to compare the first couple are mostly to show abrration will be removed by luma guide just as you point out but ewa sharpest also fails slighty.
https://slow.pics/c/cz99Yrm1

super xbr from 25 to 150 because it can do pretty much every sharpness a user could want: https://slow.pics/c/72JEh0DC

i use 100 at 1080p images but for 1080p upscaled to 4K i'm using the 150 version.

i was also planning to add a couple of measured differences but that was a waste of time:
madVR bilinear beats all mpv results... by 2 to 3 PSNR same for super xbr while all mpv results are pretty much the same depending on the channel... i left one measurement in the zip archive with all the source files.
some more modern matrix also only do Y which i could cheat but that's to much work for me. i may add some luma test and i will definitely do some performance test because mpv can do that unlike madVR where is have to estimate.

https://drive.google.com/file/d/1E7-XBHJ6Jlndafm5URsxCU3z-oAmZolW/view?usp=sharing

edit: performance benchmark

profile=gpu-hq
vo=gpu-next
gpu-api=vulkan
screenshot-high-bit-depth=no
deband=no
dither=no
audio=no
untimed=yes
video-sync=display-desync
vulkan-swap-mode=immediate
opengl-swapinterval=0
osd-msg1="FPS: ${estimated-display-fps}"

4060
this didn't work at all as i hoped the task is so trivial the noise in the data is insane
ravu r2 1300 to 1600 FPS ~60 % GPU load at 2730 mhz
ravu r3 1100 to 1350 FPS ~66 % GPU load at 2730 mhz
old super xbr shader 800-1450 63% GPU load at 2730 mhz terrible result
perdition 1100-1600 40-60 % GPU load at 2730 mhz
joint 1000-1500 60 % GPU load at 2730 mhz
cscale jinc 1100-1600 FPS 53% GPU load at 2730 mhz
cscale bilinear 1000-1700 60 % GPU load at 2730 mhz

this is pointless.

madVR
super xbr
4 shader
1.25 ms
1.25 ms
1.5 ms
0.49 ms x shift

6 ms for the rest
10.5 ms 95 fps...
at 255 mhz (not a typo)

a fast image upscale check:
https://slow.pics/c/KbViAj68
the mpv one is not the standard mpdn version where is was ported from madVR and mpdn are very similar if not the same.
ewa lanczossharp is the version from profile=gpu-hq which is without anti ringing i guess letting it look like madVR jinc is just better both are most likely the same.
ravu is just performing bad compared to super xbr in this image letting me yet again assume i'm not using it correctly.

both mpdn and madVR super xbr perform well while the mpv version is just subpar.

PSNR are other matrix are pointless there is x shifting super XBR doesn't need a x shift for image scaling.
source image https://artoriuz.github.io/blog/images/mpv_upscaling/antiring/reference.png

edit even more:
super xbr and ravu r2 are both faster then profile=gpu-hq for 1080-2160.
mpv super xbr 720-780 fps 83 % GPU load
ravu r2 ~800 fps 81 % GPU load
profile=gpu-hq 690-720 fps 83% GPU load

i'm speechless