javascript - Using AWS SDK (S3.putObject) to upload a Readable stream to S3 (node.js) - Stack Overflow

admin管理员组

文章数量:1392110

My goal is to upload a Readable stream to S3.

The problem is that AWS api seems to accept only a ReadStream as a stream argument.

For instance, the following snippet works just fine:

const readStream = fs.createReadStream("./file.txt") // a ReadStream

await s3.putObject({
    Bucket: this.bucket,
    Key: this.key,
    Body: readStream,
    ACL: "bucket-owner-full-control"
}

Problem starts when I try to do the same with a Readable (ReadStream extends stream.Readable).

The following snippet fails

const { Readable } = require("stream")
const readable = Readable.from("data data data data") // a Readable

await s3.putObject({
    Bucket: this.bucket,
    Key: this.key,
    Body: readable,
    ACL: "bucket-owner-full-control"
}

the error I get from AWS sdk is: NotImplemented: A header you provided implies functionality that is not implemented

*** Note that I prefer a Readable rather than a ReadStream since I'd like to allow passing streams that are not necessarily originated from a file - an in-memory string for instance. So a possible solution could be converting a Readable to a Readstream to work with the SDK.

Any help will be much appreciated!

My goal is to upload a Readable stream to S3.

The problem is that AWS api seems to accept only a ReadStream as a stream argument.

For instance, the following snippet works just fine:

const readStream = fs.createReadStream("./file.txt") // a ReadStream

await s3.putObject({
    Bucket: this.bucket,
    Key: this.key,
    Body: readStream,
    ACL: "bucket-owner-full-control"
}

Problem starts when I try to do the same with a Readable (ReadStream extends stream.Readable).

The following snippet fails

const { Readable } = require("stream")
const readable = Readable.from("data data data data") // a Readable

await s3.putObject({
    Bucket: this.bucket,
    Key: this.key,
    Body: readable,
    ACL: "bucket-owner-full-control"
}

the error I get from AWS sdk is: NotImplemented: A header you provided implies functionality that is not implemented

*** Note that I prefer a Readable rather than a ReadStream since I'd like to allow passing streams that are not necessarily originated from a file - an in-memory string for instance. So a possible solution could be converting a Readable to a Readstream to work with the SDK.

Any help will be much appreciated!

• javascript
• node.js
• amazon-s3
• stream

Share Improve this question Follow edited Apr 27, 2022 at 18:10 Tamir Nakar asked Apr 27, 2022 at 17:00 Tamir NakarTamir Nakar 1,07311 gold badge1111 silver badges1818 bronze badges 6

• 1 Did you happen to search for that exact error text? There seems to be a number of similar question that provide possible solutions. Part of the issue is that a string of data data data data is not a file and a readable stream from that does not have any kind of metadata about its ContentType or similar. – Alexander Staroselsky Commented Apr 27, 2022 at 18:49
• Just tried to add a ContentType ='STRING_VALUE' but get the same behavior. According to the docs, the putObject method gets Readable ReadableStream Blob string Uint8Array or Buffer as body – Tamir Nakar Commented Apr 27, 2022 at 19:56
• 1 ContentType according to the documentation is defined as "A standard MIME type describing the format of the object data.". Which is defined by w3 standards. This would be values such as text/csv or text/plain for example. putObject cannot infer if the string of data data data data is a text file or csv file or similar. – Alexander Staroselsky Commented Apr 27, 2022 at 20:13
• The STRING_VALUE was taken from one of their examples. Anyway, just tryied text/csv and got the same :/ – Tamir Nakar Commented Apr 27, 2022 at 20:32
• 1 It's not clear what else you have tried, but the header es from also content length not being present. Try adding ContentLength as well. – Alexander Staroselsky Commented Apr 27, 2022 at 20:55

 |  Show 1 more ment

2 Answers 2

Sorted by: Reset to default 4

Given the Readable is no longer a file that has metadata such as MIME type and content length you'd need to update your putObject to include those values:

const { Readable } = require("stream")
const readable = Readable.from("data data data data") // a Readable

await s3.putObject({
    Bucket: this.bucket,
    Key: this.key,
    Body: readable,
    ACL: "bucket-owner-full-control",
    ContentType: "text/plain",
    ContentLength: 42 // calculate length of buffer
}

Hopefully that helps!

Example with length calculated and using Buffer:

import { Readable } from "stream";

  saveCreativeImage(name: string, image: Buffer): Promise<string> {
    const options: PutObjectRequest = {
      ACL: 'bucket-owner-full-control',
      Bucket: EnvConfig.S3_CREATIVES_BUCKET_NAME,
      Key: name,
      Body:  Readable.from(image),
      ContentType: 'image/png',
      ContentLength: image.length
    };

Note you used to be able to send Buffer directly but now I think a StreamingBlobTypes is required. Which is defined here: https://github./awslabs/smithy-typescript/blob/21ee16a06dddf813374ba88728c68d53c3674ae7/packages/types/src/streaming-payload/streaming-blob-mon-types.ts#L22

I think this was changed here: https://github./aws/aws-sdk-js-v3/mit/96938415e65ccc4353be8f816e919215de12a1b7#diff-252b2f214c37b3c487fd068bff4968eaa1c8a6085dc9eba0d7bfe15239b05094

本文标签： javascriptUsing AWS SDK (S3putObject) to upload a Readable stream to S3 (nodejs)Stack Overflow