File upload works only with standard read streams

[ovk]

I've been trying to upload a file (using multipart POST request) that is inside TAR archive (I use tar-stream library to get stream to a particular file inside TAR archive). But it always fails with ECONNRESET error. After a bit of experimenting it seems like it can only upload streams constructed directly with fs.createReadStream and fails with irrelevant error for all other streams.

Here's a minimal example to reproduce problem. If PassThrough stream is passed into formData then the following error is occurred:

ERROR { Error: socket hang up
    at createHangUpError (_http_client.js:254:15)
    at Socket.socketOnEnd (_http_client.js:346:23)
    at emitNone (events.js:91:20)
    at Socket.emit (events.js:185:7)
    at endReadableNT (_stream_readable.js:974:12)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9) code: 'ECONNRESET' }

Server:

var http = require('http');

http.createServer(function (request, response) {                                                                                    
	request.on('data', function(chunk) {
      console.log(chunk.length);
    });    
    request.on('end', function() {
      response.writeHead(200, "OK", {'Content-Type': 'text/html'});
      response.end();
    });                                                           
}).listen(8077);

Client:

var request = require('request');
var fs = require('fs');
var PassThrough = require('stream').PassThrough;

var rs = fs.createReadStream('some_file.bin'), pt = new PassThrough();	
rs.pipe(pt);	
request({
	'method': 'POST',
	'url': 'http://127.0.0.1:8077',
	'formData':	{
		'uploaded_file': pt
	}
}, function (err, httpResponse, body)
{
	if (err)
		console.log('ERROR', err);
	else
		console.log({ 'statusCode': httpResponse.statusCode, 'body': body });
});

[philipp-spiess]

I experience the same problem when I try to wrap a Buffer in a read stream to upload it as a valid multipart request.

[sam0x17]

I have also been trying to find a way around this issue for some time and have experienced the same issues mentioned above. Has anyone found a workaround for this? Perhaps this is actually a bug with the form-data package?

[PierreCavalet]

Same problem here, I have a base64 string and I need to send it as a file. I convert my base64 to a buffer and then, just like @philipp-spiess I wrap it in a read stream, but it doesn't work. Did you guys find a work around ?

[sam0x17]

update: for now my workaround is using bhttp (https://www.npmjs.com/package/bhttp) which uses streams2

[PierreCavalet]

I've managed to upload my file using the buffer directly. You just need to specify file-related information (file name, type, ...)

var formData = {
  'file': {
    value: buffer,
    options: {
      filename: file.filename,
      contentType: file.mimetype
    }
  }
}

https://github.com/form-data/form-data#alternative-submission-methods

[ovk]

@PierreCavalet, I tried this way but unfortunately it didn't work.

My "workaround" was simply not to use request library and implement multipart POST using nodejs http service.

[philipp-spiess]

@PierreCavalet Is your buffer already encoded text? For me the buffer happens to be a binary blob which broke the encoding of the request (since it was apparently not properly encoded).

Edit: Apparently no special encoding is necessary, maybe just a wrong bit offset?! Anyway I recall that the error was the server complaining about an invalid byte.

[PierreCavalet]

@philipp-spiess my buffer is created with:

Buffer.from(base64string, 'base64')

So yes it is a binary blob. The documentation says that it creates a new Buffer containing the given JavaScript string. If provided, the encoding parameter identifies the character encoding of string.

[philipp-spiess]

@PierreCavalet The Buffer I was using was straight from multer, a file upload middleware for express.

[PierreCavalet]

@philipp-spiess This is weird. My base64 string is coming from a base64 encoding of a buffer, and this buffer is straight from my file upload. (The encoding is done by an other service, thats why we "encode and decode" for "nothing"). So it should work for you too.

[bchr02]

@SamKelly thank you so much for your recommendation of using bhttp. It works perfectly!

[sam0x17]

@bchr02 np!

[fomojola]

Tracked this error down: turns out to be an issue with form-data's _length_retriever function. If you have a stream that isn't one of the 3 fully understood streams (basically files with an fd field, HTTP responses or request output streams) you basically have to lie to form-data to get it to do the right thing (and you need to have the precise byte length of the stream). To get it to work, I ended up doing this:

var buffer = Buffer.from(base64string, 'base64');
buffer['httpVersion'] = true;
buffer.headers = { 'content-length': 64 }
var formData = {
  'file': {
    value: buffer,
    options: {
      filename: file.filename,
      contentType: file.mimetype
    }
  }
}

Very important: you MUST specify the filename field in the options, otherwise you get another exception.

[joedski]

If you can specify the file options, you should be able to specify options.knownLength with the byte length of your file stream and it will get picked up by FormData.

It's mentioned in FormData's readme though not really publicized much. Request does not mention it anywhere right now so far I can tell.

[fomojola]

Ah, that worked: teach me to do any debugging at 2am. I could have sworn I tried knownLength and got an exception, but that may have been the lack of the filename field causing chaos.

[sam0x17]

this doesn't work for my situation, because I am uploading something that is part of a gzip stream, therefore I do not know the length.

[joedski]

@sam0x17 Indeed it won't as, so far as I can tell, request doesn't currently support transfer-encoding: chunked, thus your use of bhttp.

See also

• The "transfer closed with outstanding read data remaining" error is not thrown when abort stream connection #2671
• Parse Error,the content-length and transfer-encoding #2380
• Request contains both content-length and 'Transfer-Encoding': 'chunked' #2488

[liutian1937]

mark

[BenjD90]

I faced the same problem, here is my workaround, using directly http module.

const streamSample = fs.createReadStream('data.csv').pipe(passThrough);

const formHTTP = new FormData();
formHTTP.append('sampleFieldText', 'text-sample');
formHTTP.append('myFile', streamSample);

const requestToSend = http.request({
    method: 'post',
    host: 'localhost',
    path: '/http',
    headers: formHTTP.getHeaders()
});

formHTTP.pipe(requestToSend);

requestToSend.on('response', (res) => {
    console.log('HTTP response', res.statusCode);
});

requestToSend.on('error', (e) => {
    console.log('ERROR HTTP', e);
});

[Ncifra]

@BenjD90
I used your workaround, but it seems that the stream doesn't get sent at all. I am using multer on the receiving end, if it is of any relevance.

[BenjD90]

@Ncifra I'm sure my workaround works, because I use it everyday on my current project.
You can test your stream on https://beeceptor.com/ ;)

[Ncifra]

I actually solved this by following: form-data/form-data#356 (comment)
Here is my original issue related to the problem: form-data/form-data#409

@BenjD90 Currently I reverted the code and can't test it since it was a very delayed feature on the project I am working. Maybe I was missing something since all that testing had created too much sparsed code. I did log the requests though, and the form-data object seemed to have the file in it, but multer didn't seem to receive it, while the other non binary data were being sent correctly.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[BenjD90]

@Stale This bug is steel present and should be fixed on day ^^

[sam0x17]

Yes agreed, it's as important now as it was back then. I've basically been waiting for this to be fixed before I use request again rather than bhttp. This is a basic feature of the streams and streams2 API and should be fully supported by request.

[yaacovCR]

See form-data/form-data#397 (comment) and node-fetch/node-fetch#707 (comment)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sam0x17]

Issue still exists in latest version

[NaNraptor]

Can confirm I am still getting it

[JoshuaRamsamooj]

I was able to solve a similar problem by setting the filename parameter. Maybe FormData is not able to deduce the filename from a passthrough stream?

var data = new FormData();
data.append('file', passThroughStream, {
  filename: 'some-filename'
});