Lightning+Lightrec updates and other optimizations #862

ZachCook · 2022-09-25T02:28:53Z

Moved the dynarec event cycles option to the emulation hacks sub-menu, fixed some bugs with event cycles to increase the max to 2048, and added SPU samples option up to 16 to batch SPU updates for improved performance.

Added hugepage, codebuffer, and mmap at 0x0 optimizations and fallbacks, see commits for more details.

Use lightrec's GTE registers directly when not using beetle interpreter (while maintaining backwards compatibility with savestates).

Removed the Cycle Timing Check dynarec option.

lgtm-com · 2022-09-25T02:43:46Z

This pull request introduces 1 alert when merging 8ae372a into bd6b9ef - view on LGTM.com

new alerts:

1 for Missing return statement

pcercuei · 2022-09-25T16:10:15Z

Makefile.common

@@ -375,7 +376,8 @@ ifeq ($(HAVE_LIGHTREC), 1)
                  $(DEPS_DIR)/lightrec/memmanager.c \
                  $(DEPS_DIR)/lightrec/optimizer.c \
                  $(DEPS_DIR)/lightrec/reaper.c \
-                  $(DEPS_DIR)/lightrec/regcache.c
+                  $(DEPS_DIR)/lightrec/regcache.c \
+                  $(DEPS_DIR)/lightrec/tlsf/tlsf.c


You shouldn't have to compile TLSF since you disabled the code buffer support (ENABLE_CODE_BUFFER=0)

ENABLE_CODE_BUFFER=1 was set in 3012b97 of this PR, there were many changes squashed together, and it could maybe use even more to make this more clear maybe?

Yes I saw it after my message. I guess this change should be squashed into 3012b97 then.

Ryunam · 2022-09-29T14:28:11Z

After testing this with a few games on my Windows 11 system, I have noticed a few Runahead-related issues:

As reported on Discord, enabling Runahead in RetroArch with either single instance or secondary instance mode and having the dynarec active seems to be triggering unpredictable crashes with this PR, at several spots. This is with the same values/settings used with current master. Also, the amount of dynarec cycles do not appear to affect the likelihood of the crashes.
Based on my testings so far, the crashes can occur either at startup, especially if one fastforwards right after launching the content, or when loading a save. The latter in particular is being triggered here consistently whenever I try to load my "Akumajou Dracula X: Gekka no Yasoukyoku" save from the memory card screen with Runahead active, which worked normally before this PR. Again, fastforwarding seems to greatly increase the chance of RetroArch and the core suddenly crashing with this setup.
If Dynarec SPU Samples is set to any value above 1 and Runahead is enabled with Single Instance mode, the audio will get noticeably choppy and distorted. Disabling Runahead or setting it to Second Instance Mode makes it sound normal again.

lgtm-com · 2022-10-01T02:00:05Z

This pull request introduces 1 alert when merging 1e71531 into bd6b9ef - view on LGTM.com

new alerts:

1 for Missing return statement

LibretroAdmin · 2022-10-01T03:05:12Z

@Ryunam Hi, are your issues addressed now?

Ryunam · 2022-10-01T11:47:23Z

Okay, I have performed a few more tests after the last commit:

The audio issue caused by the combination of SPU Samples set to a value higher than 1 and Runhead Single Instance mode seems to be fixed. I tried switching the values back and forth during gameplay and everything appears to sound normally.
However, the Dynarec SPU Samples by itself (with or without Runahead) seems not to play nicely with a few games. "Final Fantasy" (the PS1 remake of FF1) stood out during my testing, as any value higher than 1 triggers missing and sustained notes in the music of that game. A clear example can be seen by starting the game, then immediately entering the first city, Corneria. Set the samples to 16 and the music will start having missing parts or oddly prolonged notes. Setting the Samples value back to 1 makes the music play normally again. I assume this is partly expected behavior, as the feature in question might not work well with certain games depending on how the music files are streamed, but i figure it's worth pointing it out.
~~Interestingly, the crashes do not seem to happen anymore either! I have tried to trigger them multiple times and so far so good.~~

ZachCook · 2022-10-01T14:36:50Z

@Ryunam
I'm surprised the crashes disappeared, though I couldn't reproduce so just let me know if they come back.

The SPU Samples option was expected to cause that, I'll add a note in the description about it, it's basically just running the sampling at a lower frequency and then running multiple times to catch up, it's a performance hack, I'm surprised it works as well as it does. Though now that you found a game I can test I might be able to improve it further by doing some run time detection or tweaking the timing code, but that may be a bigger project for the future.

Ryunam · 2022-10-01T16:08:28Z

Sorry @ZachCook, I stand corrected. The crashes appeared once again while I was testing other games, so they seem to be unrelated to the SPU changes.
They are also very unpredictable: sometimes the affected games will launch normally and then crash when loading a save, sometimes the crash will occur almost immediately at startup, some other times they won't happen at all.

One consistent situation where I can reproduce the crash right now is with the Japanese PS1 port of Street Fighter Zero 2:

Under Core Options -> Emulation Hacks, set Skip BIOS to ON.
Have Runahead enabled with Second Instance ON in a per-core override, to make it activate automatically upon launching the game.
The amount of Runahead frames does not seem to matter. I'm using the default value of 1 in this case.
This was tested with both the glcore and Vulkan video drivers in RA.
Launch Street Fighter Zero 2 and hold the Fastforward button right after starting the game. One thing that may also contribute to the crash being triggered is pressing/mashing the Circle button while it's fastforwarding, but this was not verified.
On my system the game and RA will crash either after the Capcom logo animation, or later on when selecting an option from the main menu (for example, "Versus Mode").

The same core as compiled from current master, before this PR, never triggers any crash with these same steps and/or settings.

LibretroAdmin · 2022-11-14T21:44:45Z

Hi there, what is the state of this PR as it stands? Are there any roadblocks?

ZachCook · 2022-11-14T23:21:49Z

Hi there, what is the state of this PR as it stands? Are there any roadblocks?

I'll update it again to latest lightrec+lightning, and see if I can reproduce the above mentioned crash, or at least narrow down what triggers it, I wouldn't really want to merge this until that is dealt with.

subrepo: subdir: "deps/lightning" merged: "800dc662" upstream: origin: "https://git.savannah.gnu.org/git/lightning.git" branch: "master" commit: "ccb8bd77" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo" commit: "2f68596"

Fix bios freeze with lightrec, AssertIRQ must change lightrec's CAUSE too cleanup unused cop0 functions

Codebuffer is much faster due to lightrec's 32-bit LUT optimization Moved lightning.h to lightning-actual.h and made lightning.h include lightning-actual.h and define HAVE_MMAP for lightning, instead of defining in the Makefile which applied it to other parts of beetle, causing issues on Windows in libretro-common/vfs

…uilds

EventCycles should work up to 2048 now, now that it is used by MDEC and Timer SPU samples option was added, audio glitches will occur in some games unless samples is 1

This allows less code to be emitted by lightrec, but will only be used if root user enables it with sysctl -w vm.mmap_min_addr=0

…htrec api This was only useful for debugging, users don't benefit from it, the config will fallback to using the beetle interpreter if it was selected The lightrec execute_one function this option called was removed, for equivalent debugging pass current timestamp instead of next_event_ts to lightrec_execute Use lightrec cop2_notify for PGXP, update interpreter to new api

subrepo: subdir: "deps/lightrec" merged: "d640c6b4" upstream: origin: "https://github.com/pcercuei/lightrec.git" branch: "master" commit: "d640c6b4" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo" commit: "2f68596"

ZachCook · 2022-11-17T21:49:48Z

Updated to recently released Lightning 2.2.0 and Lightrec 0.6

Still working on crashes with runahead and loading, the hack I had there (restart entire dynarec and switch to interpreter for a while) looks awfully suspect and I'm working on a proper fix to at least rule that out as a cause if it doesn't just fix it.

lgtm-com · 2022-11-17T21:59:06Z

This pull request introduces 1 alert and fixes 1 when merging c41b8a7 into f8a904e - view on LGTM.com

new alerts:

1 for FIXME comment

fixed alerts:

1 for FIXME comment

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

LibretroAdmin · 2022-11-19T14:08:42Z

@ZachCook Is this ready for merging now? Will this have any known regressions vs. the current state of things?

ZachCook · 2022-11-19T22:20:28Z

@ZachCook Is this ready for merging now? Will this have any known regressions vs. the current state of things?

@LibretroAdmin It is not ready yet, I am seeing lots of crashing now than can be triggered by any kind of run-ahead or loading a save state.

You could apply just 8814bcb for now, it is quite a useful improvement by itself.

LibretroAdmin · 2022-11-20T21:00:13Z

OK, I cherry-picked that commit for now and pushed it to master

pcercuei reviewed Sep 25, 2022

View reviewed changes

LibretroAdmin marked this pull request as ready for review September 25, 2022 18:29

ZachCook added 14 commits November 17, 2022 15:41

lightrec: update include files and folder

8df60c2

lightrec: Adjust to lightrec API changes and HAVE_MMAP for lightning

3bb725c

Fix bios freeze with lightrec, AssertIRQ must change lightrec's CAUSE too cleanup unused cop0 functions

Change GTE to allow using lightrec's registers

91e458c

lightrec: attempt memfd and hugepages for ram mmap

82d0ae7

add cpu_timestamp hack to so lightrec tracks more cycles

70a6828

improve lightrec debugging by allowing threaded recompiler in debug b…

6f5e72d

…uilds

clean and optimize loop around lightrec's execution functions

be36b39

Fix eventcycles, add SPU Samples, allows much better performance

8814bcb

EventCycles should work up to 2048 now, now that it is used by MDEC and Timer SPU samples option was added, audio glitches will occur in some games unless samples is 1

support MainRAM at 0x0, use INVALID_PTR instead of NULL

3ad17a1

This allows less code to be emitted by lightrec, but will only be used if root user enables it with sysctl -w vm.mmap_min_addr=0

Fix Windows and cleanup error handling in init_mmap

ffa8400

git subrepo pull deps/lightrec

c41b8a7

subrepo: subdir: "deps/lightrec" merged: "d640c6b4" upstream: origin: "https://github.com/pcercuei/lightrec.git" branch: "master" commit: "d640c6b4" git-subrepo: version: "0.4.3" origin: "https://github.com/ingydotnet/git-subrepo" commit: "2f68596"

ZachCook force-pushed the libretro-dynarec-backup branch from 1e71531 to c41b8a7 Compare November 17, 2022 21:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lightning+Lightrec updates and other optimizations #862

Lightning+Lightrec updates and other optimizations #862

ZachCook commented Sep 25, 2022

lgtm-com bot commented Sep 25, 2022

pcercuei Sep 25, 2022

ZachCook Sep 25, 2022

pcercuei Sep 25, 2022

Ryunam commented Sep 29, 2022

lgtm-com bot commented Oct 1, 2022

LibretroAdmin commented Oct 1, 2022

Ryunam commented Oct 1, 2022 •

edited

ZachCook commented Oct 1, 2022

Ryunam commented Oct 1, 2022 •

edited

LibretroAdmin commented Nov 14, 2022

ZachCook commented Nov 14, 2022

ZachCook commented Nov 17, 2022

lgtm-com bot commented Nov 17, 2022

LibretroAdmin commented Nov 19, 2022

ZachCook commented Nov 19, 2022

LibretroAdmin commented Nov 20, 2022

Lightning+Lightrec updates and other optimizations #862

Are you sure you want to change the base?

Lightning+Lightrec updates and other optimizations #862

Conversation

ZachCook commented Sep 25, 2022

lgtm-com bot commented Sep 25, 2022

pcercuei Sep 25, 2022

Choose a reason for hiding this comment

ZachCook Sep 25, 2022

Choose a reason for hiding this comment

pcercuei Sep 25, 2022

Choose a reason for hiding this comment

Ryunam commented Sep 29, 2022

lgtm-com bot commented Oct 1, 2022

LibretroAdmin commented Oct 1, 2022

Ryunam commented Oct 1, 2022 • edited

ZachCook commented Oct 1, 2022

Ryunam commented Oct 1, 2022 • edited

LibretroAdmin commented Nov 14, 2022

ZachCook commented Nov 14, 2022

ZachCook commented Nov 17, 2022

lgtm-com bot commented Nov 17, 2022

LibretroAdmin commented Nov 19, 2022

ZachCook commented Nov 19, 2022

LibretroAdmin commented Nov 20, 2022

Ryunam commented Oct 1, 2022 •

edited

Ryunam commented Oct 1, 2022 •

edited