admin管理员组

文章数量:1334133

I am trying to configure AWS Firehose to write to a specific s3 directory. For this, in the prefix section of Firehose console, I gave "Sample/". Attaching the screenshot.

Even after this, in the S3, its creating the folder structure S3bucket/Sample/YYYY/MM/dd/file.json.

Is there a way to omit the expressions (eg.YYYY/MM/dd) and let the firehose use only the prefix, like this => S3bucket/Sample/file.json?

I am trying to configure AWS Firehose to write to a specific s3 directory. For this, in the prefix section of Firehose console, I gave "Sample/". Attaching the screenshot.

Even after this, in the S3, its creating the folder structure S3bucket/Sample/YYYY/MM/dd/file.json.

Is there a way to omit the expressions (eg.YYYY/MM/dd) and let the firehose use only the prefix, like this => S3bucket/Sample/file.json?

  • amazon-web-services
  • amazon-s3
  • amazon-kinesis
  • amazon-kinesis-firehose
Share Improve this question asked Nov 20, 2024 at 15:31 hari krishnanhari krishnan 133 bronze badges
Add a comment  | 

2 Answers 2

Reset to default 0

One solution is to add lambda function that triggers on S3 events and changes the final file name. In your case removing the date and other stuff and final name like, file.json

You can avoid the default time partitions using the dynamic partitioning. That will override the default partitions.

Only caveat is that the dynamic partitioning should be enabled while creating the firehose stream.

You can refer the documentation on how to enable it. https://docs.aws.amazon/firehose/latest/dev/dynamic-partitioning-partitioning-keys.html

本文标签： amazon web servicesConfiguring AWS Kinesis Firehose to a specific S3 FolderStack Overflow

版权声明：本文标题：amazon web services - Configuring AWS Kinesis Firehose to a specific S3 Folder - Stack Overflow 内容由网友自发贡献，该文观点仅代表作者本人， 转载请联系作者并注明出处：http://www.betaflare.com/web/1742349000a2458106.html， 本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌抄袭侵权/违法违规的内容，一经查实，本站将立刻删除。