Segmentation Fault: 11, large async.queue size

[rtran9]

What version of async are you using?
v4.4.7

Which environment did the issue occur in (Node version/browser version)
async@2.0.1

What did you do? Please include a minimal reproducible case illustrating issue.

var async = require("async");

var count = 0;
var queue = async.queue(function(item, callback) {
    count += 1;
    console.log(count);
    callback(null);
});

queue.drain = function () {
    console.log("done");
};

var data = new Array(20000);
queue.push(data);

What was the actual result?

Node throws this error Segmentation fault: 11 after popping some of the items from the queue.

...
13440
13441
13442
13443
13444
13445
13446
13447
13448
13449
13450
13451
Segmentation fault: 11

To get this to run in the first place without a Maximum call stack size exceeded error, I increased the stack size: node --stack-size=32000.

Can anyone explain to me whats going on? It works up until a certain array size.

[hargasinski]

See #1265 for a little more info.

Essentially, queue expects the worker to call callback asynchronously. As mentioned in that issue, you can use async.ensureAsync or async.setImmediate for your case to ensure that callback is called asynchronously.

Using async.setImmediate, your code would change to:

var async = require("async");

var count = 0;
var queue = async.queue(function(item, callback) {
    count += 1;
    console.log(count);
    async.setImmediate(callback, null);
});

// rest of your code... 

or using async.ensureAsync:

var async = require("async");

var count = 0;
var queue = async.queue(async.ensureAsync(function(item, callback) {
    count += 1;
    console.log(count);
    callback(null);
}));

// rest of your code...

In terms of a little more detailed explanation of what's happening to answer your question. The call stack basically manages memory for a program; it can only hold a maximum number of functions. Every time you call a function synchronously, you add it to the stack. When you call another function from that function, you again add it to the top of the stack and so on. The functions don't get removed until they return (it's a little more complicated then that, but that's the basic idea). If the stack runs out of space, an error is thrown.

When worker in the queue calls callback (adding it to the stack), the callback invokes the worker function again with the next task (adding it to the stack). And worker will again call callback and so on. Since these are synchronous functions, they will be waiting for the return of callback before returning, meaning that they wouldn't get removed from the stack. Eventually, the stack runs out of space, and the error is thrown. By wrapping callback in setImmediate, the worker doesn't have to wait for callback to return, so it will be able to get removed from the call stack, freeing up space on it.

[rtran9]

@hargasinski thank you! this explains it

[alexellis]

@caolan @hargasinski I cannot manage to make this work, I still get a Segmentation fault: 11 with 36k files:

Can you suggest a pattern or alternative to get past this?

$ node --stack-size=42000 app.js ../flat/

'use strict'

let fs = require('fs')
let async = require('async')
let request = require('request')

var q = async.queue((task, cb) => {
    console.log(task);
    async.setImmediate(cb(), null);
}, 5);

q.pause();
fs.readdir(process.argv[2], (err, res) => {

    for (var i = 0; i < res.length; i++) {
        q.push(res[i]);
    }
    q.resume();
})

[hargasinski]

@alexellis I think it might be because you're invoking cb before passing it to async.setImmediate. It should be async.setImmediate(cb, null);. See if it stills give you an error after that.

Also, quick side note, q.push accepts an array of tasks, so instead of looping over res, you can just do q.push(res). See the QueueObject docs for a little more info on it.

[alexellis]

How embarrassing that I couldn't see those (). Thanks very much and for mentioning that push supports arrays. Keep up the good work.

This is something I put together to help people learn the async library: https://github.com/alexellis/async-example

[hargasinski]

Awesome, thanks for putting that together, I'll check it out.