Too many open files.... Is it possible to increase the chunk size from 100 to say... 1000?

[remmi11]

Number of features for service parcels: 1562692
Getting chunks of 100 features, will make 15627 total requests
events.js:183
throw er; // Unhandled 'error' event

Error: EMFILE: too many open files, open 'c:\Users{user}\Desktop\AGStoShapefile\shapefiles\parcels\partials\8190.json'

[tannerjt]

In index.js, you can try and replace every instance of '100' with '1000'. This should break up the features by chunks of 1000 instead of 100.

[bkaplan1]

When I change the 100 to another number like 500 or 1000 in the index.js file, I get the following error. The REST services indicates one can get 1000 records at a time. Any suggestion for a fix? Thanks

Number of features for service Parcels: 239469
Getting chunks of 500 features, will make 479 total requests
internal/streams/legacy.js:57
throw er; // Unhandled stream error in pipe.
^

Error: Invalid JSON (Unexpected "!" at position 1 in state STOP)
at Parser.proto.charError (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\agsout\node_modules\jsonparse\jsonparse.js:90:16)
at Parser.proto.write (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\agsout\node_modules\jsonparse\jsonparse.js:267:27)
at Stream. (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\agsout\node_modules\JSONStream\index.js:21:12)
at Stream.stream.write (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\agsout\node_modules\through\index.js:26:11)
at Request.ondata (internal/streams/legacy.js:15:31)
at Request.emit (events.js:182:13)
at IncomingMessage. (C:\Users\XXXXX\AppData\Roaming\npm\node_modules\agsout\node_modules\request\request.js:1080:12)
at IncomingMessage.emit (events.js:182:13)
at IncomingMessage.Readable.read (_stream_readable.js:486:10)
at flow (stream_readable.js:922:34)
at resume (_stream_readable.js:904:3)
at process._tickCallback (internal/process/next_tick.js:63:19)

[uswoods]

@bkaplan1 Same for me:

Number of features for service TX: 426614
Getting chunks of 1000 features, will make 427 total requests
internal/streams/legacy.js:59
      throw er; // Unhandled stream error in pipe.
      ^

Error: Invalid JSON (Unexpected "!" at position 1 in state STOP)
    at Parser.proto.charError (/usr/lib/node_modules/agsout/node_modules/jsonparse/jsonparse.js:90:16)
    at Parser.proto.write (/usr/lib/node_modules/agsout/node_modules/jsonparse/jsonparse.js:267:27)
    at Stream.<anonymous> (/usr/lib/node_modules/agsout/node_modules/JSONStream/index.js:21:12)
    at Stream.stream.write (/usr/lib/node_modules/agsout/node_modules/through/index.js:26:11)
    at Request.ondata (internal/streams/legacy.js:16:26)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/usr/lib/node_modules/agsout/node_modules/request/request.js:1080:12)
    at emitOne (events.js:116:13)
    at IncomingMessage.emit (events.js:211:7)
    at IncomingMessage.Readable.read (_stream_readable.js:475:10)
    at flow (_stream_readable.js:846:34)
    at resume_ (_stream_readable.js:828:3)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickCallback (internal/process/next_tick.js:180:9)

[Ualas]

Has anyone found a workaround to this error?

[mgottholsen]

Try a smaller chunk size. I had this same problem, and changed it from 100 to 200 and was able to retrieve from my endpoint.

[emilyrbowe]

Okay, bumping this issue for those on the thread: did anyone figure out the solution here?

[emilyrbowe]

And...I'm endorsing @mgottholsen's solution — 250 was the magic number for me. Thinking this issue is closed.