amazon web services - AWS Datasync S3 to Google Cloud Storage: Full PutObjectFailed errors - Stack Overflow

admin管理员组

文章数量:1345400

Hope you are doing great.

We have a daily datasync job which is orchestrated using Lambdas and AWS API. The source locations are AWS S3 buckets and the target locations are GCP cloud storage buckets. However recently we started getting an error on datasync tasks (It worked fine before) with a lot of failed transfers due to the error "S3 PutObject Failed":

[ERROR] Deferred error: s3:c68 close("s3://target-bucket/some/path/to/file.jpg"): 40978 (S3 Put Object Failed) 

We use manifest to backup files. I didn't change anything in IAM roles etc. I don't understand why It just stopped working. Some S3 PUT works but the majority fail

Did anyone run into the same issue ?

Hope you are doing great.

We have a daily datasync job which is orchestrated using Lambdas and AWS API. The source locations are AWS S3 buckets and the target locations are GCP cloud storage buckets. However recently we started getting an error on datasync tasks (It worked fine before) with a lot of failed transfers due to the error "S3 PutObject Failed":

[ERROR] Deferred error: s3:c68 close("s3://target-bucket/some/path/to/file.jpg"): 40978 (S3 Put Object Failed) 

We use manifest to backup files. I didn't change anything in IAM roles etc. I don't understand why It just stopped working. Some S3 PUT works but the majority fail

Did anyone run into the same issue ?

• amazon-web-services
• amazon-s3
• google-cloud-platform
• aws-datasync

Share Improve this question Follow asked yesterday AlexisAlexis 7544 bronze badges

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

I've encountered similar problems with S3 PutObject failures in cross-cloud transfer jobs. Here are some things to check:

• Quota limits - Check if you've hit any AWS or GCP quotas. This can happen if your data volume increased or if other processes are using the same resources.
• IAM token expiration - Even though you didn't change IAM roles, check if your tokens or credentials might be expiring during the job. This is common with long-running transfers.
• Network connectivity - Verify there haven't been any changes to VPC settings, security groups, or network policies between your environments.
• File size or format issues - Are the failing transfers related to specific file types or sizes? Sometimes large files need different transfer settings.

In general, look at the full CloudWatch logs for your Lambda functions to see if there are more detailed error messages before the "S3 Put Object Failed" error.

本文标签： amazon web servicesAWS Datasync S3 to Google Cloud Storage Full PutObjectFailed errorsStack Overflow