Skip to content

Convert a JSON array of objects to line-delimited JSON without parsing the entire array into memory

License

Notifications You must be signed in to change notification settings

danmactough/to-ndjson

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

to-ndjson

Convert a JSON array of objects to line-delimited JSON without parsing the entire array into memory.

If you need to parse very large JSON that is unfortunately formatted as an array of objects, this is your jam.

Usage

All options of Readable and Writable streams can be passed to the ToNDJSON constructor, but the most relevant option is readableObjectMode. If you set readableObjectMode: true, your consumer will receive a stream of JSON objects. Otherwise, the default behavior is that your consumer will receive a stream of JSON stringified objects, each followed by a newline.

const { ToNDJSON } = require('to-ndjson');
const { pipeline } = require('stream');
const fs = require('fs');
const { EOL } = require('os');

pipeline(
  fs.createReadStream(someHugeJsonArray),
  new ToNDJSON({ readableObjectMode: true }),
  async function* filter(lines) {
    for await (const line of lines) {
      if (line.property.match(/some test/)) {
        yield JSON.stringify(line) + EOL;
      }
    }
  },
  fs.createWriteStream(someOutputFile),
  (err) => {
    if (err) {
      console.error(err);
    } else {
      console.error('Done');
    }
  }
);

Prior art

This module depends on creationix/jsonparse by Tim Caswell and was inspired by dominictarr/JSONStream. In fact, the only reason to use this module instead of JSONStream (which can do the same thing and more) is to get compatibility and interoperability with the current Node streams ecosystem, such as stream.pipeline.

License

(The MIT License)

About

Convert a JSON array of objects to line-delimited JSON without parsing the entire array into memory

Resources

License

Stars

Watchers

Forks

Packages

No packages published