SevenZipCompressor Compressing callback returning significantly wrong progress #67

Mgamerz · 2019-10-13T16:57:19Z

I am not sure if this is an issue with sevenzipsharp or the 7z.dll (I am using one from 7z 19.00), but when I subscribe to the Compressing event, when I inspect the argument's PercentDone value, it seems to be wildly off. It will go from 0-100 very quickly (for a 2GB source directory I am compressing), in which I believe it is filling buffers. After this reaches 100%, I get no more progress notifications until the archive job is complete - which can take several minutes.

When I use 7zip gui to compress this archive, I see correct progress, so I am wondering if maybe the compression callback is binded to the wrong interface to the dll? It shows bytes written going all the way up to the 2GB of data, but it seems more like that's what's read and not what's actually been processed, as my process sits at 14GB of memory for almost 3 minutes until the archive is done being generated.

My code to generate an archive:

var compressor = new SevenZip.SevenZipCompressor();
compressor.CompressionLevel = CompressionLevel.Ultra;
//compressor.CustomParameters.Add("s", "on"); //can't figure out how to make it do solid archives
compressor.Compressing += (a, b) =>
{
    Debug.WriteLine(b.PercentDone);
    mainWindow.UpdateBusyProgressBarCallback(new ProgressBarUpdate(ProgressBarUpdate.UpdateTypes.SET_VALUE, (int)b.PercentDone));
};
compressor.FileCompressionStarted += (a, b) => { Debug.WriteLine(b.FileName); };
compressor.CompressFileDictionary(archiveMapping, archivePath);
compressor.CompressionMode = CompressionMode.Append;
compressor.CompressionLevel = CompressionLevel.None;
compressor.CompressFiles(archivePath, new string[]
{
    Path.Combine(ModBeingDeployed.ModPath, "moddesc.ini")
});
Utilities.HighlightInExplorer(archivePath);

After doing some investigation, it seems the process for archive compression works like this;

Read source files into memory (this is what the Compressing % callbacks show)
7z.dll does it's thing with the file for significant amount of time
7z.dll/SevenzipWrapper write the new memory-archive to disk. I can see bytesWritten being raised but it is not being passed back to the sevenzipcompressor it seems (or maybe I missed something).

It is step 2 that would be most useful. I am looking into seeing how it is done in 7z UI, maybe it is exposed via some sort of interface.

The text was updated successfully, but these errors were encountered:

Mgamerz · 2019-10-13T17:33:59Z

After investigating the source code, it looks like we might want to look at SetCompleted from IArchiveUpdateCallback. I have it printing out and I am getting constant byte updates from here - I imagine this is amount of archive generated at that point.

I am also getting some progress when it's outputting to disk too from that call - perhaps this is the overall archive percentage. I'm not sure what the values mean (they seem to not be specifically bytes).

This is in COM.cs. I will continue to investigate.

Edit: It seems SetCompleted is called during all 3 steps I listed above, but it is different values than the BytesWritten (which is what the compressing callback is looking at). I'm not sure what it is, but I am going to test seeing what the SetTotal() and SetCompleted() methods combined show me. Maybe 7z is precalculating the amount of operations or something it has to do (like reading source files in chunks, compression passes, and disk writes in chunks) and that is how it shows the progressbar.

Mgamerz · 2019-10-13T18:17:33Z

Sorry for the barrage of comments and emails. It is indeed SetTotal and SetCompleted. I have extended my implementation of SevenZipSharp to have a OnProgressing() callback, which uses the SetTotal (sets a _total variable in the ArchiveUpdateCallback object). When SetCompleted is called, it raises this event. I can submit some code - I am not fully sure of the proxy styling + disabling fast compression without events. I added a DetailedProgressEventArgs that returns the Total and the Amount Completed (both ulong). Having progress as a byte between 0 and 100 is not as useful as having a large value I can work with, especially when it can take several seconds between a full jump in percentage.

I will prepare some code for a PR - I expect it will take a fair amount of work to massage it into the code style for this repository. I made it a new event instead of Compressing() as I did not want to break backwards compatibility, but I am sure there is a better way to do it.

Mgamerz · 2019-10-23T05:57:22Z

Haven't forgotten about this. I am still building my app and want to test it there first before I put in a PR.

VisualUser · 2019-11-30T04:42:30Z

Hi, do you think this could be related to this:
#77

Mgamerz · 2019-11-30T04:52:51Z

Could be. I'm still planning on making PR to this repo with this code. Essentially, if you want accurate progress, you need to setup a OnProgressing() handler, as it's the actual progress values. the progress events are already part of the library but they don't seem to be used. (SetCompleted() and SetTotal() in the archiveupdatecallback and... archivecompressingcallback? files)

Not sure what this library is hooked up to, it's like... running at full capacity of buffers or something (like when 7z is using max memory it will use during an operation). I'll see if I can make PR this weekend to this repo for this feature.

If you look at how 7z behaves when you compress a large archive, you can see it goes through a lot of files quickly and then just says "compressing X" for a long time. This library seems to be hooked up to that logic rather than what powers the progress bar.

VisualUser · 2019-11-30T06:13:12Z

Ok, thanks for the information. I will try using the OnProgressing handler.

Mgamerz · 2019-11-30T15:49:12Z

You have to make to make it yourself. Not too hard since the library is fairly well structured. Otherwise you'll have to wait for PR.

…

On Fri, Nov 29, 2019, 11:13 PM VisualUser ***@***.***> wrote: Ok, thanks for the information. I will try using the OnProgressing handler. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#67?email_source=notifications&email_token=AAU4VFC6V57324S7CMPBUZ3QWH77RA5CNFSM4JAHXXUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFP2KAQ#issuecomment-559916290>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAU4VFC5FGFZEVKSCDKEEFLQWH77RANCNFSM4JAHXXUA> .

Lukiluc29 · 2023-06-25T06:33:05Z

@Mgamerz @VisualUser Hello, do you have any news on this problem?

squid-box added the bug Something isn't working label Oct 13, 2019

Mgamerz linked a pull request Nov 30, 2019 that will close this issue

More accurate compressing and decompressing percent #78

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SevenZipCompressor Compressing callback returning significantly wrong progress #67

SevenZipCompressor Compressing callback returning significantly wrong progress #67

Mgamerz commented Oct 13, 2019 •

edited

Mgamerz commented Oct 13, 2019 •

edited

Mgamerz commented Oct 13, 2019

Mgamerz commented Oct 23, 2019

VisualUser commented Nov 30, 2019

Mgamerz commented Nov 30, 2019 •

edited

VisualUser commented Nov 30, 2019

Mgamerz commented Nov 30, 2019 via email

Lukiluc29 commented Jun 25, 2023

SevenZipCompressor Compressing callback returning significantly wrong progress #67

SevenZipCompressor Compressing callback returning significantly wrong progress #67

Comments

Mgamerz commented Oct 13, 2019 • edited

Mgamerz commented Oct 13, 2019 • edited

Mgamerz commented Oct 13, 2019

Mgamerz commented Oct 23, 2019

VisualUser commented Nov 30, 2019

Mgamerz commented Nov 30, 2019 • edited

VisualUser commented Nov 30, 2019

Mgamerz commented Nov 30, 2019 via email

Lukiluc29 commented Jun 25, 2023

Mgamerz commented Oct 13, 2019 •

edited

Mgamerz commented Oct 13, 2019 •

edited

Mgamerz commented Nov 30, 2019 •

edited