Reading Partial Zip content

[TJ-Edwards]

Hello,

I am having an issue with parsing a partial zip request. My scenario is that I need to be able to read images out of the archive, but due to the size I want to retrieve the content in chunks. However, I believe I am missing something. Whenever I try to process my blob, I get the following error.

//blob
size: 1124376968
type: "application/zip"

//Error
ZipExtractor.ts:23 ERROR Error: Uncaught (in promise): File format is not recognized.

Currently I am just retrieving a random byte range based on the archive size. This is a very temporary approach to see if partial reads could be accomplished. I realize I might be cutting files off because I dont have the central directory. I plan on getting that from an external source and not the library itself. Does anyone see this as being a potential problem?

For a bit more context I am using the ZipExtractor class provided in the repo for parsing the blob. I did some messing around to force the blob type to see if that would help, but that did not work out.

//ZipExtractor.ts
export interface ZipContent {
[filePath: string]: {
name: string
content: string
}
}

export class ZipExtractor {
public extract(zipData: Blob): Promise {
zipData = zipData.slice(0, zipData.size, "application/zip");
let zipContent: ZipContent = {};
return new Promise((resolve, reject) => {
zip.createReader(new zip.BlobReader(zipData), (zipReader) => {
let beginDate: Date = new Date();
console.info(zip reader opened at ${beginDate}, zipData);

            zipReader.getEntries(async (zipEntries: zip.Entry[]) => {
                console.info(`${zipEntries.length} entries read`, zipData);

                try {
                    let i = 0;
                    for (let zipEntry of zipEntries) {
                        if (zipEntry.filename.includes(".jpg")) {
                            let entryContent: string = await this.readEntryContent(zipEntry);
                            zipContent[zipEntry.filename] = {
                                name: zipEntry.filename,
                                content: entryContent
                            };
                            console.info(`#${++i} => ${zipEntry.filename}`);
                        }
                    }

                    zipReader.close(() => {
                        let endDate: Date = new Date();
                        console.info(`zip read ended at ${endDate} - ${endDate.getTime() - beginDate.getTime()}ms`);
                        resolve(zipContent);
                    });

                } catch (e) {
                    zipReader.close(() => {
                        reject(e);
                    });
                }
            });
        }, reject);
    });
}

private readEntryContent(entry: zip.Entry): Promise<string> {
    return new Promise<string>((resolve, reject) => {
        entry.getData(
            new zip.BlobWriter('text/plain'),
            (resultText) => {
                resolve(URL.createObjectURL(resultText));
            },
            (current: number, total: number) => {
                console.debug(`${entry.filename}: ${current}/${total}B`);
            });
    });
}

}

Any guidance here would be greatly appreciated! Thank you for your time.

[gildas-lormeau]

You're using the legacy version of zip.js (i.e. < 2.x) which has been abandoned more than 4 years ago. You have to update zip.js to the last version.

[TJ-Edwards]

I figured out what my issue was regarding the version. I had old types installed. I am facing a new issue though. Whenever I try to get entries on a partial zip download I get the following error: "Uncaught (in promise): Error: End of central directory not found" (line below).

const entries = await reader.getEntries();

Basically I am creating a image viewer for large archives so the user doesnt have to download the entire zip to view its content. So I figured I would ask for a gig of content at a time and process the images out of that batch. Does your library support that and if so do you have any samples I could reference?

Thanks for your time.

[gildas-lormeau]

zip.js does not support reading entries without reading first the central directory located at the end of the zip file. This is deliberate, because it's the only way to reliably read a zip file. However, you could maybe use a instance of HttpRangeReader in order to read a zip file from the network without downloading it entirely.

[TJ-Edwards]

Is it possible to set up a HTTP Request with a range header using that class? I have a test that seems to be making a ton of HTTP Request for the file. Below is my sample. Ideally I would like to pass a Byte Range and ask for 500mb at a time, so I can download and process a file in chunks. If it is possible can you reply with a sample? What is the purpose of chunkSize in the configuration below?

loadZip(source){
        zip.configure({ chunkSize: 256, useWebWorkers: true });
        const zipReader = new zip.ZipReader(new zip.HttpRangeReader(this.source, { useXHR: false }));
        const entries = await zipReader.getEntries();
        await this.parseZip(entries);
}

public async parseZip(entries) {
        let zipContent: ZipContent = {};
        if (entries.length) {
            let i = 0;
            for (let zipEntry of entries) {
                if (zipEntry.filename.includes(".jpg") && i <= 5) {
                    const dataBlobWriter = new zip.BlobWriter();
                    zipEntry.getData(dataBlobWriter, (resultText) => {
                        console.log(URL.createObjectURL(resultText));
                    });
                    console.info(`#${++i} => ${zipEntry.filename}`);
                }
            }
        }
    }

[gildas-lormeau]

Remove zip.configure({ chunkSize: 256, useWebWorkers: true }); or set chunkSize to a higher value like 512 * 1024 * 1024. FYI, it's documented here: https://gildas-lormeau.github.io/zip.js/api/interfaces/Configuration.html#chunkSize. The API documenation is generated from this definition file that you should use if you're writing TS code.

[gildas-lormeau]

BTW, the call to zipEntry.getData is incorrect, it does not accept a callback function as 2nd parameter anymore since the version 2.x but returns a Promise, see https://gildas-lormeau.github.io/zip.js/api/interfaces/Entry.html#getData.getData-1.