Zlib Memory Consumption

[blamishaw]

Hi,

I'm using archiver to append JSON data to a write stream, archive it in a tarball (~1.7GB), and gzip it. However, when I run my code, the memory consumption of the process goes up proportional to the size of the archive. When I exec gzipping to the shell, the process takes less than 1/3 of the memory used by archiver. Here's a snippet of the relevant code:

const writeStream = fs.createWriteStream(archivePath);
writeStream.on('warning', (err) => console.log(err));
writeStream.on('error', (err) => console.log(err));

const finalizeOnClose = (archive: archiver.Archiver) => {
  return new Promise<void>((resolve, reject) => {
    writeStream.on('close', () => {
      console.log(`Archive successfully created at ${archivePath}`);
      resolve();
    });
    archive.finalize().catch((err) => reject(err));
  });
};

const archive = archiver('tar', { gzip: true, zlib: 9 });
archive.pipe(writeStream);

for (const [i, json] of jsons) {
   archive.append(JSON.stringify(json)), {
      name: `file${i}.json`,
    });

(async () => await finalizeOnClose(archive))

I was able to recreate this issue by downloading files of various sizes from here and the memory footprint was consistently proportional to the size of the file when using zlib in archiver. Why might this be?