option for update data removal

[eadmaster]

I'd like to suggest adding an option to remove the firmware update data as it is available in CisoPlus.

Some users may also like an option to remove the video files in order to save more space.

[unknownbrackets]

Overall, I've aimed to make more "lossless" optimizations than lossy ones.

I'm not against update removal. I'm not sure what CisoPlus does, but I guess the right thing would be to write out the sectors as zeroes.

Removing video/audio worries me, as it can cause things to break. Maybe it would be possible to overwrite the MPEG stream with highly-compressible blank data, which would kill the video itself, but reduce the chances games or other things breaking unexpectedly.

-[Unknown]

[Danixu]

Would be nice to remove update data, but like unknownbrackets says: converting it into dummy files instead remove it. I think that some games stop working if there's no update files.

About video... I preffer the full games, because sometimes you loss usefull videos to save space on memory stick, and today you can buy 64GB SD cards for 20€ and an adapter with two slots by 3€ (128Gb for 43-45€).

Greetings!!

[VGkav]

I would support this. Right now, I open the iso in UMDGen, delete the UPDATE folder and save as a new iso. Then run maxcso on it.

It would be very useful if there was an option to simply omit the UPDATE folder altogether.

[VGkav]

I just found out that in the game "Hot Shots Golf - Open Tee 2" deleting the UPDATE folder makes the PSP (and the PPSSPP emulator) crash on load. I had to Dummy the files inside and dax the whole thing, now it works.

[yoshi314]

i would like this to be extended to custom files.

some ps2 games have big garbage files just for padding. and those are not zero-filled sometimes. simply treating their regions as zero filled would be good enough.

Haunting Ground on PS2 has 1.7 GB of filler data it can do without, for instance. I've rebuilt the image without it and it works just fine.

[unknownbrackets]

It's also possible for an ISO to have data that isn't addressable as a file. PSP or PS2 games can of course access this data, still, so there's no sure way to confirm it's not necessary. It'd also alter the CRC, complicating verification that your disc data isn't damaged.

In theory, update data for PSP games could be considered "well known" and part of a dictionary, reducing actual CSO size. This would mean it's not zeroed out, but it's minimized in the compressed file. It's harder to do that for PS2 games, though.

-[Unknown]

[monyarm]

A good solution for update files (to allow the iso to be recreated losslessly), would be to dummy the update files, but also compress and copy them to some folder (maybe in .local/share), and reinsert those files when recreating the iso. Many games have the same update, so it would still give savings, and would be pseudo-lossless.
Alternately, maxcso could ship with a folder full of the different update files, and dummy any files in the cso that match.

[Danixu]

It is not possible to ship all the updates with the maxcso, because there are several big updates that will increase the size a lot. Also if I am not wrong, is not fully legal to provide these update files.

Can be a way to do it, but you have to know the exact update file which was removed from the ISO or will not be lossless, so a good approach can be to have a hash of all the updates to be able to detect the removed update. Then at compression time add the version to the generated dummy file as binary which will use the first two bytes of the new created file, and at decompression time get those two bytes and try to find the related file with that version in the maxcso folder (for example 371.pbp).
Also to extract the file if not exists in the maxcso folder at compression time can be a good idea.

[mirh]

Hot Shots Golf has a LBA check, so maybe you don't even actually need dummies for that.
But after some scavenging I could find a guy claiming that even some kind of content check may be possible (go figure).
I wonder how many permutations are out there, and if you couldn't use the official EBOOT.PBP on sony's servers to help here.

[unknownbrackets]

I suspect almost all the "LBA check" cases were actually just optimizations interfering with rebuilding the ISO. Many PSP games have the exact LBAs of files hardcoded in the ELF or even in a file on the disc. This was just to skip looking up the files which was apparently slow. If the files move, the game breaks - not because of any "anti-tamper" thing, but just because the pre-built file lookup table wasn't updated.

That's why I talked about zeroing out the sectors above. That wouldn't change the LBAs.

Anyway, a very common case for people having trouble running games in PPSSPP is that their ISOs are corrupt (for various reasons, truncated, transferred over a bad cable, whatever.) We calculate a CRC of the disc (which of course works for CSO too by reading the decompressed blocks) and this gives the user a way to tell. Removing update data would mean the CRC calculates to a different value, making it harder for someone to verify their disc image isn't corrupt.

Having a dictionary of update data could work, although having the actual bytes might be problematic (copyrights.) However, this would require any reader of the format (e.g. PPSSPP) to also know about the dictionary in order to calculate CRCs that match expected data, and it would actually be a slight "lie."

Of course, we could store the "expected" CRC as metadata or something, but a lot of the damaged ISO cases above are actually hacked discs, undubs, or other things the user doesn't realize they're even using which cause crashes at some point in the game. My point is it's useful to know you have a clean copy, and removing update data - even if it's restorable - complicates that.

-[Unknown]

[mirh]

Yeah, I had also come through one such case while quickly searching for the stuff above (i.e. in under 20 minutes).
So it seems kinda desperate to introduce so much complexity just to save 20MB each time
Maybe with a big collection, that may pile up to steal considerable space.. but still?

Or perhaps, could it be possible to keep the UPDATE folder 100% uncompressed inside the image, so that "concerned people" could use whatever solid compression to sub-archive their images (hoping that smarter compressors can detect duplicates, if any?)
Or maybe we have always been looking at this from the wrong angle - maybe the right way was always to reason at the level of the whole dumps folder, rather than individual iso files?

[monyarm]

It's not always just the update folder. I've freed up GBs by dummying DUMMY files, unused files, cut content, demos for other games. The UPDATE folder is simply the one that would be easiest to rebuild into an iso.