Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Idea: Genesis Everdrive <-> MiSTer save conversion #80

Open
birdybro opened this issue Feb 6, 2022 · 71 comments
Open

Idea: Genesis Everdrive <-> MiSTer save conversion #80

birdybro opened this issue Feb 6, 2022 · 71 comments

Comments

@birdybro
Copy link

birdybro commented Feb 6, 2022

https://github.com/superg/srmtools - Source code for an existing converter in c++

Source code from there. Would be nice for us in the community to link people to your site when they ask about this. :)

@euan-forrester
Copy link
Owner

euan-forrester commented Feb 7, 2022

Thanks for this!

It's definitely nice to have source code, and it looks really simple (just packing/unpacking uint8s as uint16s).

I'm not sure how it fits into the site right now: so far it's been about making available saves found around the Internet by putting them into raw form, which is then loadable by many emulators and/or writable to original hardware.

I'm wary of an explosion in formats if it starts including formats for every specific emulator/flash cart/mister core without having a plan to manage the complexity (which I don't have at the moment!) and make a nice UI. Already the nav is pretty goofy looking. I've had some ideas about using a sample save to automatically add/remove a header (which I'm guessing is a common difference between these sorts of formats), but that wouldn't cover this case.

I've been maintaining a list of emulators which are incompatible with each other in the readme (https://github.com/euan-forrester/save-file-converter#emulators-with-incompatible-save-formats), to serve as food for thought and examples when it comes time for this. I'll add this one too.

So it's an interesting usecase for sure. I'm just not sure what to do about it right now.

@euan-forrester
Copy link
Owner

euan-forrester commented Feb 25, 2022

@birdybro I've received other requests for this, so I've put together a preliminary page: https://savefileconverter.com/#/mister and it does conversions just for the Genesis.

It's not linked from the main nav because it's just in a preliminary stage and not ready for prime time yet.

Can I get your help improving it?

I don't have a MiSTer: can you tell me if any conversion is required for saves the following platforms:

  • NES
  • SNES
  • N64
  • GameCube
  • Game Boy
  • Game Boy Color
  • Game Boy Advance
  • Game Gear
  • Sega Master System
  • Sega Saturn
  • Dreamcast
  • Turbografx-16
  • PlayStation 1
  • PlayStation 2

And, what is the correct file extension for the MiSTer for saves for each of those platforms? Are they all .sav, or are some different?

These are the main platforms I'm interested in for the site right now, so I'd like to get them all filled in correctly. If there's others that require conversion, then we can talk about adding them for sure.

Thanks again for making an issue here to let me know about this need, and especially for linking to the code! That made my job a lot easier :).

@euan-forrester
Copy link
Owner

euan-forrester commented Mar 1, 2022

Also, I've noticed that Genesis games that use EEPROM to save do not need this conversion. However, what about the Genesis games that use FRAM to save? Is there someone who can test this for me on a MiSTer?

Edit: This appears to just be Sonic 3. In an emulator, it requires the same byte expansion thing as SRAM saves, but I'm not sure about a MiSTer.

@birdybro
Copy link
Author

birdybro commented Mar 2, 2022

Sorry about the delayed response!

EEPROM's are currently not implemented in the Genesis MiSTer core (doing EEPROM in FPGA takes a bit of time to do apparently and it's not been a major priority). There are SRAM hacks available for the relevant few EEPROM Genesis games so they should be compatible with SRAM conversion if that is a concern.

Regarding FRAM, I don't believe Sonic 3 is treated any differently in the Genesis core, usually for that it would need a special quirk defined with the game. I believe all of the quirks are just to allow the games to boot. There is only one SRAM quirk I am aware of that has to do with how to treat the SRAM data itself (which also allows the game to run correctly), which is just for one Sonic 1 hack that is using an incorrect SRAM initialization (0x00 instead of 0xFF which was sega's unusual spec).

https://github.com/MiSTer-devel/Genesis_MiSTer/blob/e0b5e2ddc4d41d0a06661a190a8fc35ebb203742/Genesis.sv#L1003

and

https://github.com/MiSTer-devel/Genesis_MiSTer/blob/e0b5e2ddc4d41d0a06661a190a8fc35ebb203742/rtl/system.sv#L922-L924

Other than that, Sonic 3 just seems to work, so I think the core must be similar to original hardware on this front where it doesn't worry about the unused sectors of the FeRAM in Sonic 3. This system doesn't use cartridges if that helps your understanding. :)

For the list from above, most of the systems are on the platform and the others will not be capable of being emulated on the platform due to hardware limitations, so that keeps it easy!:

NES - https://github.com/MiSTer-devel/NES_MiSTer
SNES - https://github.com/MiSTer-devel/SNES_MiSTer
N64 (won't ever come to the platform, beyond what the hardware can do)
GameCube(won't ever come to the platform, far beyond what the hardware can do)
Game Boy - https://github.com/MiSTer-devel/Gameboy_MiSTer
Game Boy Color - https://github.com/MiSTer-devel/Gameboy_MiSTer
Game Boy Advance - https://github.com/MiSTer-devel/GBA_MiSTer
Game Gear - https://github.com/MiSTer-devel/SMS_MiSTer
Sega Master System - https://github.com/MiSTer-devel/SMS_MiSTer
Sega Saturn (Work in Progress, coming soon, wishing for Sergey's safety in Ukraine...)
Dreamcast(won't ever come to the platform, beyond what the hardware can do)
Turbografx-16 - https://github.com/MiSTer-devel/TurboGrafx16_MiSTer
PlayStation 1 - https://github.com/MiSTer-devel/PSX_MiSTer (almost complete, already being used by players)
PlayStation 2 (won't ever come to the platform, far beyond what the hardware can do)

That should cover it for the list. Full list of current cores here --> https://birdybro.github.io/MkDocs_MiSTer/cores/console/

The only ones I see missing from the list are MegaCD (not sure what would need to be converted), WonderSwan, Atari Lynx, and apparently a few fringe homebrew for Atari 2600 and 7800 have save functionality but that's just hiscore weirdness and I doubt any conversion would ever be needed for that, and probably not much demand :P

I will provide you files of saves when I get home from work tonight later on, sorry about the delays again!

As for incompatibilities off the top of my head from other cores and famous emulators, I believe there were only 2 games that had weird sram copy protection checks for some reason that made it so the sram size was incorrect on save, but that was fixed a couple years ago.

@euan-forrester
Copy link
Owner

euan-forrester commented Mar 2, 2022

No worries about the delay!

Thanks for the info about Genesis EEPROM and FRAM saves -- that's super helpful! I'm definitely a bit lost in the MiSTer world, so I really appreciate this help.

For Genesis specifically, for SRAM saves I have someone testing for me and they found that apparently the saves have to be padded out with 0xFF to 64kB and won't be loaded otherwise. I'm curious because the code you linked above for converting Genesis saves doesn't seem to do that. Any insight there? I updated this tool to do that padding (TBD is to undo the padding for converting from MiSTer, once it's verified that the padding works).

And thank you for the info about the various cores. I'm glad there's lots of overlap with my list and the actual list :). And good that there's not many on the actual list beyond what I was thinking. Edit: I updated the list in the tool

Do you have any info about what conversion, if any, is required for transferring saves to each of those cores? What file extension does each one use? Are they all .sav?

Edit: Sorry I missed that you said you were going to provide files this evening!

@birdybro
Copy link
Author

birdybro commented Mar 2, 2022

Yes 0xFF is based upon Sega's standard. Genesis-Plus-GX uses it and a few other Genesis emulators that I know of. There were a few emulators that don't that are older. The genesis MiSTer core should work fine with incorrectly padded saves though last I checked. I helped make the SRAM00_Quirk, which required switching to 0xFF padding by default for the whole core, and then adding the quirk to switch to 0x00 padding only for Sonic 1 Remastered (which only need it initialized to 0x00 to work funnily enough), and it didn't impact anybody's saves retroactively as far as I have been aware. :).

The conversion to MiSTer ought to be 0xFF padding by default as the target I think, just because that is how original hardware was supposed to be. The only exception known is Sonic 1 Remastered, a weird hack :P

Thanks!

@euan-forrester
Copy link
Owner

euan-forrester commented Mar 2, 2022

Ah I see -- it seems like there was a miscommunication which resulted in the wrong file being used and thus the incorrect conclusion that padding was necessary. Thanks for clarifying that!

And yeah, I'm happy enough to leave the padding in there for converting to mister, even if it's not required. I'll add the code to un-pad a mister file before converting it back to emulator format, because naively byte expanding all the padding would result in a strange file.

I really appreciate all this help -- this stuff is a bit confusing and definitely moreso when I can't test for myself so there's extra opportunity for mistakes and misunderstandings.

I'm looking forward to grabbing those other files and taking a look through them to compare with emulator files and see what the differences (if any) are! Thank you!

@birdybro
Copy link
Author

birdybro commented Mar 2, 2022

Yeah, there were no games that failed to work with the incorrect padding other than Star Trek TNG that we found, but this was only when the save file didn't exist yet and you had the game try to look for a save. When you change the sram initialization to 0xFF, it fixes it. It seems like the Genesis SRAM stuff is not very picky at all on padding of existing saves.

@birdybro
Copy link
Author

birdybro commented Mar 2, 2022

PSX = No different than the other standard emulators. It's just the Bleem/MCD format, and I think there is only one old format that was different.

https://github.com/MiSTer-devel/PSX_MiSTer/raw/main/memcard/empty.mcd - blank psx mcd so you can see default formatting, this is identical to how Memcardrex, mednafen, and pcsx-r treat it, so you shouldn't need anything for PSX.

SNES saves worked when I pulled them off a retrode and interchanged them with bsnes before, so I don't think there is any problem there.

saves.zip - here's just a big zip of all my saves on my MiSTer for you now so you can see.

@euan-forrester
Copy link
Owner

Oh nice, so you've even tested a bunch of genesis games with respect to padding -- that's awesome to hear!

Yeah, looks like the psx file is the same format as emulators, just with a .mcd extension rather than .mcr like I've seen elsewhere. I'll make the tool just pass the file through untouched but change the extension.

And I'll take a spin through that zip file -- thank you. I'll let you know if there's a platform that's missing or if I need more info about something. Thank you!

@birdybro
Copy link
Author

birdybro commented Mar 2, 2022

Yeah MCD is the format Bleem!, memcardpro, and a couple other things use. No problem! :)

@euan-forrester
Copy link
Owner

Yeah one thing I need to do with this tool is to do a pass through the whole thing and make it more relaxed about extensions -- there's just too many cases where different extensions mean the same thing.

@Kitrinx
Copy link

Kitrinx commented Mar 3, 2022

the genesis "padding" is what logically happens if you read the 16 bit bus from an 8 bit ram chip. The chip itself is storing and returning 8 bit values, but the bus has 8 data bus lines (presumably) pulled up to 0xFF. I think this is where the original padding came from in save files. Maybe the emulator authors who did this were anticipating some theoretical use of the other half of the data bus that never happened. RAM prices being what they are, it's safe to say that games of the day weren't using sram twice as large to hold half empty space.

@birdybro
Copy link
Author

birdybro commented Mar 3, 2022

It was in an official Sega technical bulletin that said cartridges manufactured should (in most cases) have their SRAM initialized to 0xff is all I meant. :)

109776795-44720000-7c03-11eb-9a21-8f42070e6eb1

There was only one game where not doing this caused a problem.

https://github.com/MiSTer-devel/Genesis_MiSTer/issues/170

@euan-forrester
Copy link
Owner

@Kitrinx heh yeah I figured it was 16 bit read of 8 bit data :). Gotta use all that 16 BIT POWER :).

@ALL man I was hoping for an easy win with NES saves, but I looked at a couple of games that I'm familiar with and see that their 8k saves on a real cart are expanded to 32k on the mister, but the remaining data is garbage rather than padding. The garbage is virtually indistinguishable from the real data (assuming that the real data is actually the first 8k of the file). Any thoughts here? I guess I'll just try padding them out to 32k.

From eyeballing a couple of SNES saves I'm familiar with, it looks like the mister ones are the same size as the ones from a cart so no padding or transformation needed there.

@birdybro
Copy link
Author

birdybro commented Mar 4, 2022

Yeah, I've been able to use SNES saves no problem from my Retrode2 dumps, and ones from bsnes as well.

NES I've never had a problem using the saves. There are some 8kb saves in there, those are nes2.0 headered roms. It's possible most of my saves are iNES headered rom's being used. I know the ginga eiyuu densetsu was a rom i had to repair the header for (with kitrinx's help) since it was incorrect in the NRS nes 2.0 database, but it got all fixed. Once it was fixed that save is 8kb. So maybe NES 2.0 headered roms leave accurate saves and iNES does not...

@birdybro
Copy link
Author

birdybro commented Mar 4, 2022

NES.zip

Yup I used NES 2.0 Headered roms and forced a save, and it created 8KB files this time.

@euan-forrester
Copy link
Owner

Hm that's interesting -- thanks for figuring that out!

Do you think I should pad all the NES saves out to 32k, since I don't know what type of ROM will be used? Does the core accept 8kB or 32kB saves regardless of which type of ROM is being used?

For now I have it padding the save, and you can check it out here if you want to test: https://savefileconverter.com/#/mister

I also added PS1 and TG16 support -- those were just plain raw files. I'm still making my way through the list, but that's where I got to today :).

@Kitrinx
Copy link

Kitrinx commented Mar 4, 2022

the core won't start mirroring the addresses until 32kb, so any file with size up to that is fine, however if it's smaller it will cause no harm, the remaining ram will just be uninitialized data which is generally harmless

@euan-forrester
Copy link
Owner

Gotcha -- thank you for the info! I'll leave it as padding out to 32k then.

I'll work through the remaining platforms in the next few days, but in the meantime I'd love any help you could give me with testing what's there: I don't have a MiSTer of my own to verify that this works.

You can find it at https://savefileconverter.com/#/mister and so far I've done NES, SNES, Genesis, TG16, and PS1

@euan-forrester
Copy link
Owner

euan-forrester commented Mar 6, 2022

Okay I looked at Gameboy and GBA today:

For Gameboy: I noticed that many saves were 8k, but there were also a lot of 32k files. Almost all 32k files were for Super Gameboy enhanced games, but not all SGB games had 32k files. Many of the 32k files had strange alternating padding of 0x00 then 0xFF and back again.

Also I saw that for Final Fantasy Adventure, the mister save was 1k but mine off my cart (via a Retron5) is 0.5k:
Final Fantasy Adventure (U) [!].srm.zip

Can someone please test that file on their mister?

I'm just going to code the GB/GBC "conversion" to pass the file straight through in both directions for now -- please let me know if they should be padded differently or do something else different!

For GBA:

The Castlevania Aria of Sorrow save in that dir is virtually empty, but it's only 32k whereas mine of my cart (via a Retron 5) is 64k. But, in mine there's data more than halfway through the file
Castlevania - Aria of Sorrow (USA) (1).srm.zip

Similar for Metroid Fusion, where my save (via a Retron 5) has data all the way to the end of its 64k whereas the mister one is 32k:
Metroid Fusion (USA, Australia) (1).srm.zip

I certainly didn't expect to see files that were "too short" from the mister! Any thoughts here? Can you try them out too on a mister?

Again for now I've just made the tool pass the file through in each direction. As always, you can check out the latest version at https://savefileconverter.com/#/mister

@birdybro
Copy link
Author

birdybro commented Mar 6, 2022

FF Adventure - That save shows up as just 512 bytes of zeros when I unzip it:

The MiSTer sav when I make a new one is 1024 bytes of 0xFF.

For GBA... I don't know if the emulator supports 64k/128k FRAM yet.

@Kitrinx
Copy link

Kitrinx commented Mar 6, 2022

Oh.. by the way, some systems save real time clock data at the end of the save data. They'll repeat it over and over. I think gb and gba do this.

@euan-forrester
Copy link
Owner

Re FF Adventure: Oh! That's strange! I just tried both downloading the one attached here, and the original .zip file I made and both unzipped into a file with data in it. Um, can you maybe try again? Very strange.

Re GBA: Castlevania is battery-backed SRAM (one of the few) and I'm not sure about Metroid other than there's no battery :).

Interesting info about the RTC -- I'll keep an eye out for that -- thanks!

@birdybro
Copy link
Author

birdybro commented Mar 7, 2022

image

Now I see it :)

@euan-forrester
Copy link
Owner

Okay cool! Have you had a chance to test it out on the MiSTer? I hope it works, otherwise I guess we pad the files out to a minimum of 1k? I think that Final Fantasy Legend 1 also had a 512B save file if I remember right.

@euan-forrester
Copy link
Owner

Sorry for the radio silence -- we've been moving so I've been swamped in box land :). I've had a chance to look at some more systems:

Sega CD: The emulator makes a file called cart.brm that's 524B and has a footer at the end. I tried the Sonic CD save from the .zip file and it's 8kB and has a similar (but longer) footer. But it doesn't seem to work straight in the emulator after renaming (does that save actually contain gameplay data, like completed levels, etc?). Any thoughts here?

Sega Master System: I made the tool pad the file out to 32kB with 0x00, since that's how the example files looked

Sega 32X: We didn't mention this system earlier, but it seems worth adding. There's no example mister saves for it -- can someone please make one (preferably with actual game data after playing for a little while). Here's a list of 32X games that can save: https://segaretro.org/Battery_backup#32X

Lynx & Jaguar: The directories for these platforms are empty -- can someone please supply a save or two here?

Sega Game Gear: There weren't any example files provided, so I just made the tool copy the file straight over. Can someone please provide an example Game Gear file (again, with actual gameplay saved)? Here's a list of Game Gear games that can save: https://segaretro.org/Battery_backup#Game_Gear

@birdybro
Copy link
Author

birdybro commented Mar 21, 2022

Sorry as well, been kinda distracted with getting our new documentation site rolled out for the MiSTer --> https://mister-devel.github.io/MkDocs_MiSTer/

Sega CD: I have no idea how the saves work in that one. Which software emulator did you try the save with? The Sega CD core was kind of a very difficult core to even make function on the MiSTer so it's possible it may need refining for things like saves. EDIT: https://segaretro.org/CD_BackUp_RAM_Cart#Maximum_capacity - since each game will create a new save, the saves could theoretically be as large as 512KB, which is hilarious.

Sega 32x: There is no 32x MiSTer core currently. ;) The Sega Saturn core is being worked on by srg320 who lives in the Donbas region in Ukraine unfortunately :( He's okay the last we heard though. If anyone would make the 32x core, it would likely be him. When the Saturn core is finished, it's possible that work could be done on a 32x core as that means a hardware model for an SH-2 processor would be finished.

Jaguar: This is a very early stage core, I'm not even sure if saves are supported yet. @Kitrinx could probably comment on that, as she just updated the core quite a bit to actually be functional.

Lynx: There were no commercial releases for Atari Lynx with save functionality. There are a tiny handful of homebrew games with it. Robert Peip never implemented the save functionality from what I can tell, as a result.

Sega Game Gear: Here ya go, I just started a save in Shining Force The Sword of Hajya and had it autosave.
Shining Force - The Sword of Hajya (USA).zip
This is also in the SMS core :P It's an SMS/SG1000/GG combo core.

@euan-forrester
Copy link
Owner

Nice -- I like that documentation. Very friendly!

Thanks for all the info about the 32x/saturn/jaguar/lynx cores. I've gone and added some notes to my file that contains the list of platforms so I won't forget!

Sega CD: I'm using the GenesisPlus emulator on my Mac -- it's the default for OpenEmu. Here's the file it made:
cart-original.brm.zip I'm feeling at a loss here. Should we give up on the platform, or is there somewhere we can go for more info?

Game Gear: Thank you for the file! It loads fine in the GenesisPlus emulator. Curiously, the emulator makes a 26kB file initially, which is than padded out to 64kB after the first save, rather than the 32kB file from the mister. Both the mister and emulator files have data up until about the same offset though. Can you please give my save a try? Shining Force - The Sword of Hajya (USA).sav.zip

SMS: Have you had a chance to test the code I added for this one?

FF Adventure: Have you had a chance to test the save I attached above in the gameboy mister core?

GBA: So it sounds like the core here isn't complete, and these 64k saves (Metroid Fusion, Castlevania: Aria of Sorrow) just aren't supported at the moment?

@euan-forrester
Copy link
Owner

euan-forrester commented Mar 21, 2022

Oh, and WonderSwan: is there any direction you can give me here? It looks like there's lots of different and strange file sizes in the directory you gave me: 2560B, 8704B, 33280B, 131584B, 262656B, 524800B -- always 0x200 more than a power of 2? I know nothing about the WonderSwan

@euan-forrester
Copy link
Owner

Heh I didn't see the edit until just now, so here's a file I created:
saveram.zip

It's 64kB long, but I can't attest to much about it. I don't have any experience with NeoGeo, but from googling I don't think it's possible to run MAME in AES (home console) mode using OpenEmu, so it's from Metal Slug but in MVS (arcade cabinet) mode. So who knows what it contains. I got to level 2 and then copied the file. In AES mode the game is supposed to save your progress after losing all your lives so you can restart at the last level you were on.

@birdybro
Copy link
Author

birdybro commented Apr 2, 2022

64KB makes sense, as that would be the MVS backup RAM internally. MiSTer is expanding to 72KB I think so it can retain the UNIBIOS settings as well as the DIPS, game saves (which MVS doesn't have), etc...

@birdybro
Copy link
Author

birdybro commented Apr 5, 2022

Heh I didn't see the edit until just now, so here's a file I created:
saveram.zip

It's 64kB long, but I can't attest to much about it. I don't have any experience with NeoGeo, but from googling I don't think it's possible to run MAME in AES (home console) mode using OpenEmu, so it's from Metal Slug but in MVS (arcade cabinet) mode. So who knows what it contains. I got to level 2 and then copied the file. In AES mode the game is supposed to save your progress after losing all your lives so you can restart at the last level you were on.

Okay, looking at it and it appears that MiSTer initializes the Save RAM to 0xFF. It is 72KB vs the provided example being 64KB. At 0x10000 (meaning just after the first 64KB) that's where the BIOS settings appear to be stored. This save all I did was change UNIBIOS from Europe+Arcade to USA+Arcade, then it creates the save file for the game:

MiSTer Blank Save with BIOS settings saved only

Then I just set the game to easy, changed to 5 lives cuz I'm a wuss, and went into the first level and died pretty early on (totally on purpose) to game over, it asked me if I wanted to save and I said yes, it had to format the memory card, and here is that file:

MiSTer AES Save

There was no difference for the first 64KB, then a bunch of differences after that:

image

So the 8KB AES save starts at 0x10000, if you need the offset. The first 64KB is for MVS I'm assuming so I'll now switch to MVS. I then had to turn on the DIP Settings option and reset the game... I turned on Blood, and then set the code number to 1337 because. Then I changed the time to 3:22 because that's what it is right now... But it didn't let me for some reason (unibios maybe? i dunno). I hit exit in the menu, it tells me to turn off dip 1 so I do... and the system reboots, but the save doesn't get updated interestingly enough. It offered to load the game despite being in MVS and Arcade mode, so I did. I died, but chose not to save this time, opened the on screen display to see if it would save, and it did. Odd. Might be high score or bookkeeping figures saved I dunno. Here's that file:

AES save but loaded on MVS and soft dip settings, maybe not saved

Now in HxD I can see that the first 64KB are populated by lots of data. So the first 64KB must be MVS and the next 8KB are AES.

image

@euan-forrester
Copy link
Owner

@birdybro hey, I'm looking at Sega CD saves and had some questions about the mister. I looked through the saves you gave me earlier, and the ones for sega cd games had just one file and it was 8kB in size and was named after the ROM.

On other platforms, there's 2 files: one for the internal memory on the sega cd itself, and another for a ram cartridge. The former is always 8kB but the latter can be different sizes.

So I'm a little confused about the mister. What happens if you make both types of save for the same game? What are the files named, and how large are they? I found that Popful Mail is a good one to use because you can select where you want to save in the main menu, and make a save quickly just by going to the first area of the game then opening the menu. Would it be possible to please send me both types of save from that game?

I also found that you had a couple of saves for Shining Force CD (3R), and the files were 512kB in size but the 'footer' in the file was only after 8kB, making the file actually able to only store 8kB of data (the rest was padded with 0s). This seems very strange to me! Any thoughts here?

Thanks again for all your help! I've got a proof of concept working for moving sega cd saves between mister/emulator/megaeverdrivepro so I'm excited to get all the details figured out and get it onto the site!

@birdybro
Copy link
Author

birdybro commented Aug 17, 2022

Thanks for all your work on this as always!

For the explanation as to why there are two size of files, this is due to a toggle in the on-screen display for the core:

https://github.com/MiSTer-devel/MegaCD_MiSTer/blob/7ff9ebc90bd97368ec751ee1322f79234bf2d876/MegaCD.sv#L329

When it's set to "Internal" then it's 8KB. When it's set to "Internal+Cart" then it's a 520KB size save file of 8KB internal combined with a 512KB external backup cartridge (because this is the theoretical limit of an original Sega CD in terms of addressing assigned, even though no Backup RAM cart released contemporaneously allowed for this). Example video of the toggle just to give you perspective:

2022-08-16.21-36-17.mp4

Basically on MiSTer h0000 --> h1FFF is the 8KB internal RAM, the remaining is reserved space for the theoretical addressable limit of 512KB for the backup cartridge option, totally 520KB for a combined save that can see both.

To test your 8KB --> 520KB save theory, I started Popful Mail using the 8KB save. Then I switched the option in the OSD to use Internal+Cart (520KB resulting file size). I go to Continue and it says no save data exists. The game also didn't ask me to initialize so it must be seeing the first file. It doesn't recognize save data unless I do something and then save I guess -_-

The reasoning behind both the files being combined is because we use a reserved index number for our save system that is standardized across the cores and it only allows for 1 save per game in order to be minimal in resource utilization.

Here's the requested saves to compare:

Popful Mail (USA) (RE).zip

I don't remember what happened with all the shining force stuff, so my testing methodology could have been incorrect and I've learned a lot more since then, so we'll just ignore that :P

EDIT: oh and the differences... At the footer of a MegaCD 8KB internal RAM section there is the following:

image

At the end of the 512KB section (end of file as well) is the following:

image

I can only guess all the values in these files are populated by the BIOS + Game, ekeeke of gpgx also thought the same in the issue you raised there when discussing how MiSTer does it, and I think he's right.

@euan-forrester
Copy link
Owner

Gotcha! That makes total sense now, and it's a nice solution to put them both into the same file.

Yeah, to resize the saves so they can transfer from emulators <-> megaeverdrivepro I had to fiddle with the bits he mentioned, and it worked perfectly.

I'm a little bummed I haven't figured out how to parse the actual files themselves yet, to pull them apart if there's multiple saves in them, but at least I can get them to be recognized by different devices.

Thanks for all that info about the mister -- I'll be able to add support for creating and pulling apart those files no problem.

Thank you for such a quick and super helpful answer!

@euan-forrester
Copy link
Owner

This is now up and running: https://savefileconverter.com/#/mister

Thank you again! Next up is making the UI for converting sega cd saves for the 2 genesis flash carts

@euan-forrester
Copy link
Owner

@birdybro I saw that the Saturn core now supports saving! I know nothing about Saturn save files, but I'd love to add support for it to the tool.

Any chance you can provide me with a few save files, and ideally give me any info you know about differences (if any) between them and save files made by emulators?

BTW, I'm not sure if you saw, but a while back I added full support for parsing and creating Sega CD save files. I'm guessing that the Saturn is going to be somewhat similar: being something of a file system containing multiple potential saves.

@birdybro
Copy link
Author

birdybro commented Aug 16, 2023

I would wait a little bit until the Saturn core is officially in the project. It's kind of in an alpha/beta state right now.

@euan-forrester
Copy link
Owner

Fair enough! Thank you! I'll keep and eye out, and also please don't hesitate to reach out if something new happens that should be added (I see some work being done on N64 as well!)

@euan-forrester
Copy link
Owner

@birdybro I noticed that recently development on the N64 core has stopped/finished? I know that during development, the author experimented with different save file formats. Can you offer any advice here about whether to include support for MiSTer N64 files on the savefileconverter site, and if they should be included what the final format ended up being?

@birdybro
Copy link
Author

birdybro commented Apr 23, 2024

@RobertPeip is the core author so he probably is the best source of information as to how the save file format for n64 works. I know the format was compatible with another existing format out there somewhat.

@RobertPeip
Copy link

Save files are just plain raw data without header for each of the types(eeprom/sram/flash/controllerpak).

If you need examples, I can provide some.

@euan-forrester
Copy link
Owner

Yes please, I'd love an example of each! Just so I can check endianness, file extensions, and any padding (e.g. are either of the 2 EEPROM sizes padded out or not).

Thanks for the super quick reply and for all your work on this core!

@RobertPeip
Copy link

saves.zip

This file should contain examples of all. If i'm missing some size, just tell me.

@euan-forrester
Copy link
Owner

Thank you! It looks like they're all there, but I'm unclear on what 007 - The World Is Not Enough (USA).sav is. It looks similar to the controller pak save for the same game, but it's longer? Can you please give me a bit more info about that one?

@birdybro
Copy link
Author

birdybro commented Apr 23, 2024

https://github.com/MiSTer-devel/Main_MiSTer/blob/master/support/n64/n64.cpp#L54-L61 and https://github.com/MiSTer-devel/Main_MiSTer/blob/master/support/n64/n64.cpp#L158-L164 - This might help point you in the correct direction.

https://github.com/ChronoAndross/n64SaveConverter - This also has some details on the complexities of different types of saves for n64 in general, but you probably have most of these represented already if you already convert n64 saves on your site.

@euan-forrester
Copy link
Owner

Thanks, yeah, it's that 007 - The World Is Not Enough (USA).sav file specifically that doesn't match any of the N64 file types that I'm familiar with. The code snippets mention TPAK which I guess is the Transfer Pak, and I see that that game supports the Transfer Pak, but I'm not familiar with what format that data is in and my controller pak code couldn't parse it.

Can I please get a bit more info about that file specifically? All the others made complete sense.

@birdybro
Copy link
Author

birdybro commented Apr 23, 2024

.sav is an externally loaded gameboy save file. The core has the option to load a gameboy rom and then it will automatically load a same-named gameboy save file. If there is a transfer pak save as a result of this made it is a .tpk file. Some games supported plugging in a gb/gbc game into the transfer pak accessory to get extra benefits in the n64 game.

mupen64plus-next handles this format similarly i believe.

@euan-forrester
Copy link
Owner

Thank you for this information!

It looks to me that 007 - The World Is Not Enough (USA).sav is much too large to be a Gameboy save file, and I don't see a corresponding .tpk or similar file in the zip.

@RobertPeip could you please give me a little info about what 007 - The World Is Not Enough (USA).sav is for in that zip you provided? Everything else in that file was straightforward, but I'm unsure what this one represents.

@RobertPeip
Copy link

Sorry it was late yesterday evening. Ignore that file. It was some old format where 4 controller paks where combined in one file with swapped endianess. This format was used in the beta phase but is not in use anymore.

It could be that some users still have them, but there is no point in still supporting them. If the file is loaded with the current version and saved again, it will be automatically converted to the new format.

@euan-forrester
Copy link
Owner

Ah I see -- that makes total sense now. Thank you!

That's great that the core will automatically convert those old files, so I won't write any code to parse them then.

Thank you again for all this!

@euan-forrester
Copy link
Owner

@birdybro Do you happen to have any Famicom Disc System saves from the MiSTer? I stumbled onto this link https://www.reddit.com/r/emulation/comments/sbpyhd/converting_srm_to_sav_a_free_and_easy_diy_method/ that seems to indicate that there's a header in the MiSTer saves that's not present in emulator saves.

Do you happen to have an example file from something like Metroid/Castlevania/Zelda/Zelda 2? It just needs enough progress to contain some non-zero data, so even just naming a slot in Zelda 1 or Zelda 2 for example.

@birdybro
Copy link
Author

birdybro commented May 5, 2024

Link no Bouken - The Legend of Zelda 2 (Japan) (Rev 1).zip

Sorry for the delay, I forgot to do this after I read it lol

@euan-forrester
Copy link
Owner

No worries, and thank you!

Hm, well I certainly see that this file doesn't run in an emulator. For starters the emulator save is a mere 0x1e bytes long and the MiSTer one is 0x20000 bytes long, and the MiSTer one contains all sorts of strings that look like game data or metadata or something.

I don't see an obvious opportunity to add a short header to the emulator file to make it line up with the MiSTer one as described in the link. I'll have to look more closely.

In the meantime, if you happen to have any other FDS saves from the MiSTer I'd love to see them. With only 0x1e bytes to work with in Zelda 2 it's hard to find any sort of pattern.

@euan-forrester
Copy link
Owner

Fantastic -- thank you! I'll dig into those and see if I can find any patterns. Presumably making an emulator file will involve slicing out a small part of the MiSTer file.

Not sure about the other direction given the amount of data in the MiSTer files, so if I can find the right spot to put the save data I'll maybe ask you to test if that's okay?

@euan-forrester
Copy link
Owner

In other news MiSTer N64 support is now live on the savefileconverter site -- thank you @birdybro and @RobertPeip !

@euan-forrester
Copy link
Owner

@birdybro I dug into the FDS saves and figured out what's going on. The MiSTer stores an entire image of the disc as its save file (plus a header & footer) whereas emulators seem to store the diffs between unwritten and written copies of the disc image in one of a variety of patch file formats (mine happens to use the UPS format, I read of others using the IPS format).

I wrote up a bunch of notes in case I need to reference them in the future, but for now this looks way too complex since:

  1. There's no standard format for emulators
  2. No one has requested this for the MiSTer page

For now, if someone requested this I'd just walk them through how to do it by hand.

Thank you though for those save files! I was hoping to be able to add this feature easily but at least it was an interesting and educational journey!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants