System compiled dxvk causes stack smashing with wine when playing yakuza-0/kiwami.

[notmentaloutlaw]

Tried playing yakuza-0 / yakuza-kiwami. Both work with wine on its own; its just when I install dxvk they don't even start and show an error message.

0120:err:seh:NtRaiseException Unhandled exception code c0000409 flags 1 addr 0x7ffffd5255c2

distribution: Gentoo Linux
gpu: RTX 4080 SUPER
gpu driver: proprietary nvidia drivers version 550.78
wine version: wine-vanilla-9.7
dxvk version: 2.3.1-r1
vulkan-loader version: 1.3.280.0

I have tried enabling the environment variables below but they didn't give me anything for some reason.

VK_INSTANCE_LAYERS="VK_LAYER_KHRONOS_validation"
DXVK_LOG_LEVEL="debug"
DXVK_LOG_PATH="/home/user/"

[Blisto91]

Post a full Wine log if you can. Maybe ran with WINEDEBUG=+loaddll

[mbriar]

Without a log, best guess would be that you probably mixed up system32 and syswow64 when installing

In a default Wine prefix that would be as follows:

export WINEPREFIX=/path/to/wineprefix
cp x64/.dll $WINEPREFIX/drive_c/windows/system32
cp x32/.dll $WINEPREFIX/drive_c/windows/syswow64
winecfg

wine version: wine-vanilla-9.7

Also you should probably know that Yakuza 0 needs esync to reliably save: ValveSoftware/Proton#492 (comment)

[doitsujin]

If there's no log despite setting DXVK_LOG_PATH then chances are our DLLs aren't even getting loaded. Both games are known to work fine with DXVK.

[notmentaloutlaw]

In order to setup dxvk in my wineprefix, which I created a brand new one just for yakuza-0 I did the following:

WINEPREFIX="<game_dir>/prefix/"
WINEARCH="win64"
WINEESYNC=1
WINEFSYNC=1
WINE_FULLSCREEN_FSR=1
WINE_FULLSCREEN_FSR_STRENGTH=2
DXVK_FRAME_RATE_LIMIT=145

setup_dxvk.sh install --symlink

Then I just replaced the setup line with wine <game_dir>/<yakuza.exe>

I will try enabling wine debugging hopefully that will help.

[notmentaloutlaw]

I just tried WINEDEBUG=+all and WINEDEBUG=+loaddll and I am getting no extra output other than what I said.

[mbriar]

That's impossible, there can't just be that one line with +all

[notmentaloutlaw]

Maybe its due to the strip useflag in gentoo I will remove it and see if it provides a log. The thing is with +all I ran it without dxvk and was getting a lot more lines of output than I usually get so I thought it was working.

[mbriar]

I don't think it should be related to the wine build config, i don't think it's possible to disable the logging completely at build time, and it seems like it works without DXVK? But I also don't see how there could possibly be only that one line in the log and nothing else with +loaddll and DXVK loaded, it should at least print a line for loading the dlls, if nothing else.

[notmentaloutlaw]

I still couldn't figure out why the debug wasn't showing anything extra. However, I think I have solved part of the issue already. It seems that when I used setup_dxvk.sh (which is shipped with gentoo repos dxvk) the symlinked dxvk dlls were not working as it should. So after I did setup_dxvk.sh install --symlink and then replaced the symlinks with the dlls that you have on github it seems that dxvk is running finally (although the framerate is less than when I used to play with my amd rx 6800 could be just nvidia drivers or something) but its a lot more than stock wine.

I am confused on this as some games i.e fallout 4 is able to use the symlinked dlls however games like yakuza 0 and fallout new vegas can't. What could be the issue? I am not using crossdev-mingw useflag so its using dev-util/mingw64-toolchain rather than crossdev.

[notmentaloutlaw]

I was trying this again today and it seems like I was able to obtain a wine log with +loaddll.

wine.log

Thanks for the wait.

[notmentaloutlaw]

looks like when using system compiled dxvk on gentoo and running yakuza creates some sort of stack smashing.

[doitsujin]

have you tried using our release binaries instead?

I'm seeing exactly zero dxvk messages in that log though so it seems to just fail horribly before reaching any of our code for some reason.

[notmentaloutlaw]

have you tried using our release binaries instead?

I'm seeing exactly zero dxvk messages in that log though so it seems to just fail horribly before reaching any of our code for some reason.

I still couldn't figure out why the debug wasn't showing anything extra. However, I think I have solved part of the issue already. It seems that when I used setup_dxvk.sh (which is shipped with gentoo repos dxvk) the symlinked dxvk dlls were not working as it should. So after I did setup_dxvk.sh install --symlink and then replaced the symlinks with the dlls that you have on github it seems that dxvk is running finally (although the framerate is less than when I used to play with my amd rx 6800 could be just nvidia drivers or something) but its a lot more than stock wine.

I am confused on this as some games i.e fallout 4 is able to use the symlinked dlls however games like yakuza 0 and fallout new vegas can't. What could be the issue? I am not using crossdev-mingw useflag so its using dev-util/mingw64-toolchain rather than crossdev.

Yes I meant I used your release binaries and it was working. However, the ones that my system compiled for some reason in these games it wasn't working which is very weird. So the release binaries on github for this version of dxvk worked for yakuza 0 etc however when I try compiling these versions of dxvk on my system and used setup_dxvk.sh it doesn't work and provides that wine.log that I have posted.

[notmentaloutlaw]

For comparison this is the wine.log when I use the dxvk binaries provided on github.

wine-log-dxvk-bin.log

If you compare lines 217 onwards for both logs you can see that this is where it differs.

[ionenwks]

Gentoo here, while we do default to -fstack-protector-strong system-wide, the default mingw toolchain we provide does not (prefer to avoid most security features given how fragile Wine can be, Edit: the implementation of the features themselves can also be like how gcc+mingw crashed with -fstack-clash-protection for a while). I believe any users affected by this is either passing -fstack-protector-strong or similar themselves in *FLAGS or using a different toolchain that does default-ssp (e.g. with crossdev tooling in gentoo)

Does indicate there may be something wrong with either dxvk or mingw64-runtime, but to have a shot at reproducing would need to build both with it (haven't tried, Edit: and I assume the binaries here do not already).

If really needed, Gentoo could work around this by forcing explicit -fno-stack-protector in the runtime+dxvk ebuilds (ignoring user settings) though.

[notmentaloutlaw]

As ionenwks stated its because mingw64-toolchain was compiled with -fstack-protector-strong. However, while this does indicate that something is "wrong" disabling this and recompiling dxvk does make it work.

[doitsujin]

Not sure how to provide any help here, things like asan unfortunately don't work with mingw and we obviously can't use native builds with Windows games.

FWIW, building dxvk itself with -fstack-protector-strong works fine here with Kiwami at least. Don't have 0 installed right now.

[ionenwks]

Not sure how to provide any help here, things like asan unfortunately don't work with mingw and we obviously can't use native builds with Windows games.

FWIW, building dxvk itself with -fstack-protector-strong works fine here with Kiwami at least. Don't have 0 installed right now.

Yeah that gave me no issues either, I needed to build mingw's runtime itself with it then build dxvk against these to start having issues. fwiw doing that doesn't seem to break wine itself built using the same mingw.

Unfortunate but if debugging is going to be hard, guess passing -fno-stack-protector is still what we can go with downstream anyhow.

[ionenwks]

Actually, I just remembered that support for this is kind of new in mingw64 11 (it used to be straight up broken with <=10). This gives decent chances of the issue being in mingw itself rather than dxvk.

Think it can be considered experimental and skipping using it seems fair in that context.