javascript - NodeJS: Uploading a remote file to S3 with request and knox - Stack Overflow

admin管理员组

文章数量:1327979

I'm trying to devise a way to upload a file from a url to s3 using request and knox. Currently, my code looks like this:

request(item.productImage, function(err, res, body) {
  if (!err && res.statusCode == 200) {
    fs.writeFile('/tmp/' + filename, body, 'base64', function(err, data){
      if (err) { 
        return console.log(err);
      }

      client.putFile('/tmp/' + filename, '/item/' + item._id + '/' + filename, function(err, res) {
        if (err) {
          return console.log(err);
        }
      }); 
    });
  }
});

This doesn't work as it downloads about 652 bytes of a 4kb file before it stops. Strangely, if I don't provide a callback to fs.writeFile() it downloads the entire 4kb locally.

What's the best way of acplishing this?

I'm trying to devise a way to upload a file from a url to s3 using request and knox. Currently, my code looks like this:

request(item.productImage, function(err, res, body) {
  if (!err && res.statusCode == 200) {
    fs.writeFile('/tmp/' + filename, body, 'base64', function(err, data){
      if (err) { 
        return console.log(err);
      }

      client.putFile('/tmp/' + filename, '/item/' + item._id + '/' + filename, function(err, res) {
        if (err) {
          return console.log(err);
        }
      }); 
    });
  }
});

This doesn't work as it downloads about 652 bytes of a 4kb file before it stops. Strangely, if I don't provide a callback to fs.writeFile() it downloads the entire 4kb locally.

What's the best way of acplishing this?

• javascript
• node.js
• amazon-s3

Share Improve this question Follow asked Feb 28, 2012 at 5:06 JamesJames 6,5091111 gold badges6161 silver badges8787 bronze badges 1

• Thanks for the question. I was about to ask the same question and I got the answer also here. – user644745 Commented Mar 3, 2012 at 3:36

Add a ment  | 

1 Answer 1

Sorted by: Reset to default 10

There are a number of questions about this here on Stackoverflow, but I can't seem to find one that answers your question. The solution below should work, however, I'm having trouble getting knox to work at all on my machine right now. I hope you will be more lucky!

UPDATE: I seem to have had some problems with s3 here, the code below works -- I did change one thing, you need to specify encoding as null to request, so you get a Buffer back. Otherwise, binary data won't work so well.

request(item.productImage, {encoding: null}, function(err, res, body) {
  if(!err && res.statusCode == 200) {
    var req = client.put('/item/' + item._id + '/' + filename, {
      'Content-Type': res.headers['content-type'],
      'Content-Length': res.headers['content-length']
    });

    req.on('response', function(res) {
      console.log('response from s3, status:', res.statusCode, 'url:', req.url);
    });

    req.on('error', function(err) {
      console.error('Error uploading to s3:', err);
    });

    req.end(body);
  }
});

Note: With this solution, you avoid having to buffer the files to disk - that's why I chose to use the lower level put method of the knox client.

本文标签： javascriptNodeJS Uploading a remote file to S3 with request and knoxStack Overflow