createWriteStream; close doesn't finish writing all data to file

[d3x0r]

Windows 7/10,
node 5.6.0, and latest builds...
This seems like something fairly critical...

"use strict";

const fs = require( 'fs' );
var outData = fs.createWriteStream( "output.txt" );
outData.write( "line one" );
outData.write( "line two" );
outData.close();

results in a file that only has 'line one' in it.
it can be several lines... I'm often missing 5 lines in this one file... but basically every file never writes all information to the file.

this might be similar to #2314

[evanlucas]

Here is a working example. In order to have a guarantee that write() has finished, you must wait for the callback on it.

'use strict'

const fs = require('fs')
const stream = fs.createWriteStream('out.txt')
stream.write('line one', () => {
  stream.write('line two', () => {
    stream.close()
  }) 
})

[d3x0r]

Okay I started to dig into the code, and added some console.log( stream ); after the writes.... there's a 'pendingcb' which goes up, and things start to get buffered... so I supposed maybe if there wasn't a callback it would fail.

It will take some time to restructure the code to fit HAVING to use the callbacks.
I would still think that close operation should pend itself after the writes, or mark the stream as closed so the last write can complete the close....

but even in a more complex case where I continue on to do other work with other files for a long amount of time the output still doesn't go... so the pending writes apparently never get called without having a callback of some sort...

[Slayer95]

@d3x0r , maybe you are looking for WritableStream#end() ? That will automatically handle closing the stream for you after pending writes are done.

Note that close() is not part of the public API (https://nodejs.org/api/stream.html#stream_class_stream_writable)

[d3x0r]

I had added

outData.write( "line two" );
outData.end();
outData.close();

but that made no difference.

[vkurchatkin]

This works as expected. close immediately closes fd without flushing ending writes. Just call end and fd will be closed automatically after writes are flushed.

[d3x0r]

confirmed; replacing close with end lets it work :)

[ORESoftware]

close() is no longer a method on the writable instance, use end() instead

[seantcanavan]

is there no way to use await or anything like that in conjunction with end() to halt the process and wait for the memory buffer to write to the file? i'm trying to email a CSV and it keeps going out without its full contents. I'm assuming because writing is still queue'd after end() and also after my sendEmail call happens. the question is why after calling end() does it take my local process exiting to get the full file output? I've verified locally that the attachment has all the correct contents. It's just while executing the file must not be finished before getting email.

EDIT: Solution is here: nodejs/help#1641 (comment) credit to @provCristianMaluenda:

        // create the output file
        const csvFileName = 'invoice_report_' + getUnixFriendlyTimestampString() + '.csv';
        const csvFilePath = '/tmp/' + csvFileName;
        const writeStream = fs.createWriteStream(csvFilePath, { flags: 'a' });
        const csvWriteStream = format({ headers: true, delimiter: ',' });

        // pipe data from the stream to the newly created file
        csvWriteStream.pipe(writeStream);```
...
...
        // write final CSV output line
        csvWriteStream.write({
            totalDollarAmount: totalDollarAmount.toString()
        });

        // close the file handle and wait for memory buffer to be written to disk
        await this.waitForStreamClose(writeStream);
        sendGrid.sendEmail(to, from, [csvFileName]); // email sends with full file content now
...
...
    async waitForStreamClose(stream: WriteStream): Promise<void> {
        stream.end();
        return new Promise((resolve) => {
            stream.once('finish', () => {
                resolve();
            });
        });
    }