javascript - Find the content-length of the stream before uploading the file or writing the file on network - Stack Overflow

admin管理员组

文章数量:1388146

I am reading the file, zipping & encrypting it and then uploading/writing on network. But I need to know the content-length of the end stream( stream returned after passing through read, zip, encryption) to make a post request.

let zlib = zlib.createGzip(),
   encrypt = crypto.cipherIV(....),
    input = fs.createReadStream('file.jpg');
function zipAndEncrypt(){
   let stream = readStream.pipe( zlib).pipe( encrypt );
   let options = {
     "stream_length":0,
     headers: { "content-type": 'image/jpeg',
                "content-length": '123456', // need to get this length 
          .....
     }
    }

// post the stream
needle( 'post', url, stream, options )
   .then( resp => { console.log( "file length", resp.body.length);})
   .catch( err => {})
}

Above code works if I enter the correct content length in headers ( in this case I knew the length ). So I need to find the length of the stream.

so far I achieved the length by :

  let chunk = [], conLength;
  stream.on( 'data', ( data ) => {
            chunk.push( data );
        } )
        .on( 'end', () => {
         conLength = Buffer.concat( chunk ).length; 
        } );

But the post request fails, SOCKET hang up error.

It looks like stream is drained or consumed as it does not emit 'data' event after finding the length using the code above.

Tried stream.resume(). But nothing works. Could you please suggest how to find the length of the stream without consuming the stream.

I am reading the file, zipping & encrypting it and then uploading/writing on network. But I need to know the content-length of the end stream( stream returned after passing through read, zip, encryption) to make a post request.

let zlib = zlib.createGzip(),
   encrypt = crypto.cipherIV(....),
    input = fs.createReadStream('file.jpg');
function zipAndEncrypt(){
   let stream = readStream.pipe( zlib).pipe( encrypt );
   let options = {
     "stream_length":0,
     headers: { "content-type": 'image/jpeg',
                "content-length": '123456', // need to get this length 
          .....
     }
    }

// post the stream
needle( 'post', url, stream, options )
   .then( resp => { console.log( "file length", resp.body.length);})
   .catch( err => {})
}

Above code works if I enter the correct content length in headers ( in this case I knew the length ). So I need to find the length of the stream.

so far I achieved the length by :

  let chunk = [], conLength;
  stream.on( 'data', ( data ) => {
            chunk.push( data );
        } )
        .on( 'end', () => {
         conLength = Buffer.concat( chunk ).length; 
        } );

But the post request fails, SOCKET hang up error.

It looks like stream is drained or consumed as it does not emit 'data' event after finding the length using the code above.

Tried stream.resume(). But nothing works. Could you please suggest how to find the length of the stream without consuming the stream.

• javascript
• node.js
• stream
• pipeline
• needle.js

Share Improve this question Follow asked Mar 28, 2019 at 23:30 timtim 32344 silver badges1414 bronze badges 4

• Can't you send it without the content-length? – Marcos Casagrande Commented Mar 28, 2019 at 23:35
• If you need the encrypted&packed file size, you need to do this first, and then post it to the server. You can store the stream in the memory, wait till the file is pressed, then get the file size & you can perform your post request with the correct content-length header – Marc Commented Mar 28, 2019 at 23:38
• Have you seen my answer? @Tim – Marcos Casagrande Commented Mar 29, 2019 at 16:24
• @MarcosCasagrande I checked your answer. Thank you. Writing it on disk and then finding the length is good idea. – tim Commented Apr 1, 2019 at 9:05

Add a ment  | 

1 Answer 1

Sorted by: Reset to default 5

If you need to send the content length, the only way to know it, is after the file has been zipped & encrypted.

So, your solution works, but only if you send the buffer, and not the stream, because you already consumed all the data from the stream. And since you already have all the chunks in memory, you might as well send it.

let chunk = [];

stream.on('data', data => chunk.push(data))
.on('end', () => {
    const buffer = Buffer.concat(chunk);
    const conLength = buffer.length;
    // Execute the request here, sending the whole buffer, not the stream
    needle(/*...*/)
});

But if your file is too big, you're require to stream it, otherwise you will reach out of memory, an easy workaround, with a little overhead, is to pipe it to a temporary file, and then send that file. That way you can know the file size before performing the request, accessing the stream.bytesWritten property or using fs.lstat.

function zipAndEncrypt(input) {
    const gzip = zlib.createGzip();
    const encrypt = crypto.createCipheriv(algo, key, iv),

    const stream = input.pipe(gzip).pipe(encrypt);


    const fileName = tmpFileName();
    const file = fs.createWriteStream(fileName)
    stream
        .pipe(file)
        .on('finish', () => {
            let options = {
                "stream_length": 0,
                headers: {
                    "content-type": 'image/jpeg',
                    "content-length": file.bytesWritten
                }
            }

            const readStream = fs.createReadStream(fileName);

            // post the stream
            needle('post', url, readStream, options)
                .then(resp => {
                    console.log("file length", resp.body.length);
                })
                .catch(err => {})
                .finally(() => {
                    // Remove the file from disk
                });
        })

}

本文标签：